Aug 04 15:56:30.794484 ip-10-0-137-14 systemd[1]: Starting Kubernetes Kubelet... Aug 04 15:56:31.183230 ip-10-0-137-14 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:56:31.183230 ip-10-0-137-14 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 15:56:31.183230 ip-10-0-137-14 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:56:31.183230 ip-10-0-137-14 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 15:56:31.183230 ip-10-0-137-14 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:56:31.184113 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.184024 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 15:56:31.189621 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189598 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:31.189621 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189615 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:31.189621 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189619 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:31.189621 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189623 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:31.189621 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189626 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:31.189621 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189630 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189633 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189636 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189640 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189643 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189646 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189649 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189651 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189654 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189656 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189659 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189662 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189664 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189667 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189670 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189672 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189675 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189677 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189680 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189682 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:31.189846 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189685 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189687 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189690 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189692 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189695 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189697 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189703 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189706 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189708 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189711 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189713 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189716 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189718 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189720 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189724 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189727 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189729 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189733 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189735 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:31.190335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189738 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189740 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189743 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189745 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189748 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189750 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189753 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189755 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189759 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189762 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189766 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189770 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189773 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189775 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189778 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189781 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189783 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189786 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189790 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:31.190792 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189794 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189797 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189800 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189803 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189807 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189809 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189812 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189814 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189818 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189821 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189825 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189828 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189831 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189833 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189836 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189839 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189841 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189844 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189846 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189849 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:31.191269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189852 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189855 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189858 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189861 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189863 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189865 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189868 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189871 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189873 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189876 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189879 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189882 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189885 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189888 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189890 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:31.191748 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.189893 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191900 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191908 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191912 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191915 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191919 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191921 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191924 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191927 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191929 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191932 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191934 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191937 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191939 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191942 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191944 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191947 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191949 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191952 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191955 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:31.192108 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191959 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191964 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191967 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191971 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191974 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191977 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191981 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191984 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191987 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191990 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191993 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191996 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.191999 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192002 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192005 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192008 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192011 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192014 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192017 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:31.192623 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192020 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192023 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192026 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192029 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192031 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192034 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192037 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192039 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192041 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192044 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192046 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192049 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192052 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192055 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192057 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192060 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192062 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192065 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192067 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192070 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:31.193092 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192072 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192074 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192077 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192080 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192082 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192085 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192089 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192091 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192094 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192097 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192101 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192103 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192106 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192108 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192111 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192114 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192117 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192120 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192122 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192125 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:31.193587 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192128 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192130 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192133 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192135 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192138 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192141 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192144 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192147 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192149 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192152 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192154 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192157 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192159 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192162 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192164 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192167 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192169 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192172 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192175 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192177 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:31.194181 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192180 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192575 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192581 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192583 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192586 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192590 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192593 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192595 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192598 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192601 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192604 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192606 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192608 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192611 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192614 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192616 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192619 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192628 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192631 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:31.194695 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192634 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192637 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192639 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192642 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192644 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192647 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192649 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192652 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192654 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192657 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192661 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192664 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192666 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192669 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192672 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192674 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192677 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192680 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192683 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192685 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:31.195182 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192687 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192690 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192693 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192695 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192698 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192700 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192704 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192708 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192711 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192714 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192716 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192720 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192723 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192725 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192728 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192730 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192733 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192735 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192740 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:31.195688 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192743 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192746 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192748 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192752 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192755 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192758 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192760 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192763 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192766 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192768 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192771 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192774 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192776 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192779 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192781 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192784 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192787 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192791 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192794 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192796 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:31.196159 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192799 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192802 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192807 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192810 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192813 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192817 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192820 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192823 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192825 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192828 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192831 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192833 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192836 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192838 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192840 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192843 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192846 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192849 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192851 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192854 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:31.196703 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192856 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.192859 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192925 2571 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192932 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192939 2571 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192944 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192948 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192951 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192955 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192959 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192962 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192965 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192968 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192971 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192976 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192979 2571 flags.go:64] FLAG: --cgroup-root="" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192982 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192984 2571 flags.go:64] FLAG: --client-ca-file="" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192988 2571 flags.go:64] FLAG: --cloud-provider="external" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192991 2571 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192995 2571 flags.go:64] FLAG: --cluster-domain="" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.192998 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193001 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193004 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 15:56:31.197190 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193008 2571 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193011 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193014 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193017 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193020 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193023 2571 flags.go:64] FLAG: --contention-profiling="false" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193026 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193029 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193032 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193035 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193039 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193042 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193045 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193048 2571 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193050 2571 flags.go:64] FLAG: --enable-server="true" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193053 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193058 2571 flags.go:64] FLAG: --event-burst="100" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193061 2571 flags.go:64] FLAG: --event-qps="50" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193064 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193067 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193070 2571 flags.go:64] FLAG: --eviction-hard="" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193073 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193078 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193081 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193084 2571 flags.go:64] FLAG: --eviction-soft="" Aug 04 15:56:31.197777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193087 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193090 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193093 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193096 2571 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193101 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193104 2571 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193107 2571 flags.go:64] FLAG: --feature-gates="" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193111 2571 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193114 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193117 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193120 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193123 2571 flags.go:64] FLAG: --healthz-port="10248" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193126 2571 flags.go:64] FLAG: --help="false" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193129 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193132 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193135 2571 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193137 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193141 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193144 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193147 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193150 2571 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193153 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193155 2571 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 15:56:31.198460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193158 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193161 2571 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193164 2571 flags.go:64] FLAG: --kube-reserved="" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193167 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193170 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193173 2571 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193177 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193180 2571 flags.go:64] FLAG: --lock-file="" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193183 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193185 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193188 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193193 2571 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193197 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193216 2571 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193221 2571 flags.go:64] FLAG: --logging-format="text" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193225 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193231 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193234 2571 flags.go:64] FLAG: --manifest-url="" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193237 2571 flags.go:64] FLAG: --manifest-url-header="" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193241 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193244 2571 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193248 2571 flags.go:64] FLAG: --max-pods="110" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193252 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193254 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193257 2571 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 15:56:31.199034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193260 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193264 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193267 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193270 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193277 2571 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193280 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193283 2571 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193286 2571 flags.go:64] FLAG: --pod-cidr="" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193289 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193295 2571 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193298 2571 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193301 2571 flags.go:64] FLAG: --pods-per-core="0" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193304 2571 flags.go:64] FLAG: --port="10250" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193307 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193311 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-091a57d945f882895" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193314 2571 flags.go:64] FLAG: --qos-reserved="" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193317 2571 flags.go:64] FLAG: --read-only-port="10255" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193320 2571 flags.go:64] FLAG: --register-node="true" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193323 2571 flags.go:64] FLAG: --register-with-taints="" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193326 2571 flags.go:64] FLAG: --registry-burst="10" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193332 2571 flags.go:64] FLAG: --registry-qps="5" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193335 2571 flags.go:64] FLAG: --reserved-cpus="" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193338 2571 flags.go:64] FLAG: --reserved-memory="" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193342 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193345 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 15:56:31.199670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193348 2571 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193351 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193354 2571 flags.go:64] FLAG: --runonce="false" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193357 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193360 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193363 2571 flags.go:64] FLAG: --seccomp-default="false" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193366 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193369 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193372 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193374 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193377 2571 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193380 2571 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193383 2571 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193386 2571 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193389 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193392 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193395 2571 flags.go:64] FLAG: --system-cgroups="" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193397 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193403 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193405 2571 flags.go:64] FLAG: --tls-cert-file="" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193408 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193415 2571 flags.go:64] FLAG: --tls-min-version="" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193418 2571 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193420 2571 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193423 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 15:56:31.200329 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193426 2571 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193429 2571 flags.go:64] FLAG: --v="2" Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193434 2571 flags.go:64] FLAG: --version="false" Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193439 2571 flags.go:64] FLAG: --vmodule="" Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193443 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.193447 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193567 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193571 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193574 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193578 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193582 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193585 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193587 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193590 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193593 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193596 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193599 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193601 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193604 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193607 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193609 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:31.200981 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193612 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193615 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193617 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193620 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193622 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193625 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193628 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193631 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193634 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193636 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193639 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193641 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193644 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193648 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193651 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193654 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193656 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193659 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193661 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193664 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193666 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:31.201533 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193669 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193671 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193674 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193676 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193679 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193682 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193684 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193687 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193690 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193692 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193695 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193697 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193699 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193703 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193706 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193709 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193712 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193715 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193717 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:31.202048 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193720 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193723 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193725 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193728 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193730 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193734 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193737 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193739 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193742 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193745 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193747 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193750 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193752 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193755 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193757 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193763 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193766 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193769 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193771 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:31.202595 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193774 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193777 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193779 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193782 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193784 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193787 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193790 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193793 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193795 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193797 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193800 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193802 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193805 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193807 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193810 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193812 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193814 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193817 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193821 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193824 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:31.203058 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193826 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193829 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193832 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193836 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.193839 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.194493 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.201592 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.201608 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.201622 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.201627 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201684 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201689 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201692 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201695 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201698 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:31.203563 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201700 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201703 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201706 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201709 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201712 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201714 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201717 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201719 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201722 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201724 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201727 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201729 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201732 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201735 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201737 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201740 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201742 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201745 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201749 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:31.203945 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201753 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201756 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201759 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201762 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201765 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201768 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201771 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201774 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201777 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201779 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201782 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201784 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201787 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201790 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201792 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201795 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201797 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201802 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201805 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201808 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:31.204429 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201811 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201815 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201818 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201821 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201824 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201827 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201829 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201832 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201834 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201837 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201840 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201842 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201845 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201848 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201850 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201853 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201856 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201859 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201862 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201865 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:31.204946 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201867 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201870 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201872 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201877 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201880 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201883 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201886 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201889 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201892 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201894 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201897 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201899 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201902 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201905 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201907 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201910 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201912 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201915 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201917 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:31.205448 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201920 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201922 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201925 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201927 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201930 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201936 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201939 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201942 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201945 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201948 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201951 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201954 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201957 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201960 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201963 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:31.205907 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.201965 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.201971 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202091 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202096 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202099 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202102 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202105 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202108 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202111 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202114 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202116 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202119 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202121 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202124 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202127 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:56:31.206417 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202130 2571 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202133 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202136 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202139 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202142 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202144 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202147 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202150 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202152 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202155 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202158 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202160 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202163 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202166 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202169 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202172 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202174 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202177 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202179 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202183 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:56:31.206804 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202187 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202190 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202192 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202195 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202198 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202218 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202223 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202226 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202229 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202232 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202234 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202237 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202240 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202242 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202245 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202247 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202250 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202252 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202255 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202257 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:56:31.207337 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202260 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202263 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202265 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202268 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202271 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202274 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202277 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202280 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202282 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202285 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202287 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202290 2571 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202293 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202295 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202298 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202300 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202304 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202307 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202309 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202312 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:56:31.208123 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202315 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202317 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202320 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202322 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202325 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202327 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202330 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202333 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202335 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202339 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202343 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202346 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202349 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202352 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202355 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202357 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202361 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202363 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202366 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:56:31.208785 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202368 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202371 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202373 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202376 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202378 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202380 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:31.202383 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.202387 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.202492 2571 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.204569 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.206001 2571 server.go:1019] "Starting client certificate rotation" Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.206121 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.206172 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 15:56:31.209269 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.208744 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 15:56:31.209849 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.209830 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 15:56:31.220011 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.219990 2571 log.go:25] "Validated CRI v1 runtime API" Aug 04 15:56:31.222036 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.222023 2571 log.go:25] "Validated CRI v1 image API" Aug 04 15:56:31.222799 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.222784 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 15:56:31.226189 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.226168 2571 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 aacb0a22-0393-4b97-8f86-20c145e061f3:/dev/nvme0n1p3 b4c76420-df98-4676-a4bf-e7c2572bb546:/dev/nvme0n1p4] Aug 04 15:56:31.226265 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.226187 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 15:56:31.232191 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.232086 2571 manager.go:216] Machine: {Timestamp:2026-08-04 15:56:31.230865427 +0000 UTC m=+0.324465666 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099248 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aeb61919ac070e1f755ab9c9fd4fe SystemUUID:ec2aeb61-919a-c070-e1f7-55ab9c9fd4fe BootID:c90114b2-36d2-4e49-ab38-20316874c9ad Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:04:e1:ec:c7:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:04:e1:ec:c7:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:9c:78:2b:19:21 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 15:56:31.232191 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.232189 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 15:56:31.232323 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.232279 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 15:56:31.235261 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.235231 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 15:56:31.235411 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.235264 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 15:56:31.236002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.235991 2571 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 15:56:31.236033 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.236005 2571 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 15:56:31.236033 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.236012 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 15:56:31.236712 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.236699 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 15:56:31.237883 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.237873 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 15:56:31.239194 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.239179 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 15:56:31.240921 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.240909 2571 state_mem.go:36] "Initialized new in-memory state store" Aug 04 15:56:31.241038 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241029 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 15:56:31.241074 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241062 2571 kubelet.go:486] "Attempting to sync node with API server" Aug 04 15:56:31.241074 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241072 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 15:56:31.241125 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241089 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 15:56:31.241125 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241102 2571 kubelet.go:398] "Adding apiserver pod source" Aug 04 15:56:31.241125 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241112 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 15:56:31.241507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241494 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 15:56:31.241570 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.241513 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 15:56:31.243759 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.243743 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 15:56:31.245279 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.245254 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 15:56:31.245338 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.245295 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 15:56:31.246077 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246060 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 15:56:31.246077 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246076 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246084 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246092 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246105 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246113 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246121 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246126 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246137 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 15:56:31.246146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246143 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 15:56:31.246366 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246152 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 15:56:31.246366 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246157 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 15:56:31.246994 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.246983 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 15:56:31.247052 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.247022 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 15:56:31.250588 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.250573 2571 server.go:1268] "Started kubelet" Aug 04 15:56:31.250894 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.250679 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 15:56:31.250894 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.250746 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 15:56:31.250894 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.250778 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 15:56:31.251494 ip-10-0-137-14 systemd[1]: Started Kubernetes Kubelet. Aug 04 15:56:31.252195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.252040 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 15:56:31.256912 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.256885 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 15:56:31.257029 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.256999 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 15:56:31.257029 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.257012 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 15:56:31.257423 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.257392 2571 server.go:310] "Adding debug handlers to kubelet server" Aug 04 15:56:31.258588 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.258566 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 15:56:31.261251 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.261234 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 15:56:31.261334 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.261254 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 15:56:31.261399 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.261333 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.261846 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.261825 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 15:56:31.261947 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.261867 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 15:56:31.261947 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.261887 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 15:56:31.262045 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262017 2571 reconstruct.go:97] "Volume reconstruction finished" Aug 04 15:56:31.262045 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262026 2571 reconciler.go:29] "Reconciler: start to sync state" Aug 04 15:56:31.262243 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262223 2571 factory.go:55] Registering systemd factory Aug 04 15:56:31.262243 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262242 2571 factory.go:223] Registration of the systemd container factory successfully Aug 04 15:56:31.262505 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262488 2571 factory.go:153] Registering CRI-O factory Aug 04 15:56:31.262566 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262508 2571 factory.go:223] Registration of the crio container factory successfully Aug 04 15:56:31.262620 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262569 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 15:56:31.262620 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262593 2571 factory.go:103] Registering Raw factory Aug 04 15:56:31.262620 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.262617 2571 manager.go:1196] Started watching for new ooms in manager Aug 04 15:56:31.263253 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.262136 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-14.ec2.internal.18c8a54a686980e7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-14.ec2.internal,UID:ip-10-0-137-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-14.ec2.internal,},FirstTimestamp:2026-08-04 15:56:31.250309351 +0000 UTC m=+0.343909590,LastTimestamp:2026-08-04 15:56:31.250309351 +0000 UTC m=+0.343909590,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-14.ec2.internal,}" Aug 04 15:56:31.263609 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.263596 2571 manager.go:349] Starting recovery of all containers Aug 04 15:56:31.271632 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.271588 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 15:56:31.271847 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.271820 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 15:56:31.274563 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.274547 2571 manager.go:354] Recovery completed Aug 04 15:56:31.278192 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.278177 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-898cm" Aug 04 15:56:31.278873 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.278861 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:31.280570 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.280542 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:31.280645 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.280580 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:31.280645 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.280590 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:31.281112 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.281097 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 15:56:31.281180 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.281113 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 15:56:31.281180 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.281129 2571 state_mem.go:36] "Initialized new in-memory state store" Aug 04 15:56:31.281180 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.281134 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:31.281891 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.281874 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:31.282053 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.281900 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:31.282053 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.281913 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:31.282382 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.282368 2571 policy_none.go:49] "None policy: Start" Aug 04 15:56:31.282382 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.282383 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 15:56:31.282541 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.282393 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 15:56:31.282801 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.282740 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-14.ec2.internal.18c8a54a6a373973 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-14.ec2.internal,UID:ip-10-0-137-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-14.ec2.internal,},FirstTimestamp:2026-08-04 15:56:31.280568691 +0000 UTC m=+0.374168930,LastTimestamp:2026-08-04 15:56:31.280568691 +0000 UTC m=+0.374168930,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-14.ec2.internal,}" Aug 04 15:56:31.283513 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.283499 2571 policy_none.go:47] "Start" Aug 04 15:56:31.286924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.286909 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-898cm" Aug 04 15:56:31.323802 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.323779 2571 manager.go:337] "Starting Device Plugin manager" Aug 04 15:56:31.323906 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.323828 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 15:56:31.323906 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.323843 2571 server.go:87] "Starting device plugin registration server" Aug 04 15:56:31.324115 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.324101 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 15:56:31.324176 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.324120 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 15:56:31.324275 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.324252 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 15:56:31.324381 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.324352 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 15:56:31.324381 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.324360 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 15:56:31.324919 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.324898 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 15:56:31.324990 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.324952 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.355100 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.354936 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 15:56:31.356412 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.356393 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 15:56:31.356516 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.356433 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 15:56:31.356516 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.356456 2571 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 15:56:31.356516 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.356492 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 15:56:31.358518 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.358501 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:31.426488 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.426443 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:31.427554 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.427532 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:31.427656 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.427577 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:31.427656 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.427594 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:31.427656 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.427622 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.435721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.435650 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.435721 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.435674 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-14.ec2.internal\": node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.452518 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.452497 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.456852 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.456825 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal"] Aug 04 15:56:31.456912 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.456899 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:31.458466 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.458450 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:31.458577 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.458477 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:31.458577 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.458488 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:31.459928 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.459915 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:31.460055 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460040 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.460090 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460085 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:31.460711 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460695 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:31.460781 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460708 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:31.460781 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460724 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:31.460781 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460735 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:31.460781 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460740 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:31.460781 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.460746 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:31.462008 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.461991 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.462055 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.462029 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:56:31.462520 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.462501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b2c461907233d4b90cae1267fdce864-config\") pod \"kube-apiserver-proxy-ip-10-0-137-14.ec2.internal\" (UID: \"2b2c461907233d4b90cae1267fdce864\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.462593 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.462529 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11916bc9c437c84f693e9d6334241cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"e11916bc9c437c84f693e9d6334241cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.462593 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.462548 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11916bc9c437c84f693e9d6334241cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"e11916bc9c437c84f693e9d6334241cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.462725 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.462712 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:56:31.462770 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.462738 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:56:31.462770 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.462755 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:56:31.481897 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.481877 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-14.ec2.internal\" not found" node="ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.486387 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.486370 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-14.ec2.internal\" not found" node="ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.553713 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.553678 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.563272 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.563243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11916bc9c437c84f693e9d6334241cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"e11916bc9c437c84f693e9d6334241cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.563272 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.563273 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b2c461907233d4b90cae1267fdce864-config\") pod \"kube-apiserver-proxy-ip-10-0-137-14.ec2.internal\" (UID: \"2b2c461907233d4b90cae1267fdce864\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.563466 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.563290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11916bc9c437c84f693e9d6334241cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"e11916bc9c437c84f693e9d6334241cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.563466 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.563338 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e11916bc9c437c84f693e9d6334241cc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"e11916bc9c437c84f693e9d6334241cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.563466 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.563345 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e11916bc9c437c84f693e9d6334241cc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal\" (UID: \"e11916bc9c437c84f693e9d6334241cc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.563466 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.563367 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b2c461907233d4b90cae1267fdce864-config\") pod \"kube-apiserver-proxy-ip-10-0-137-14.ec2.internal\" (UID: \"2b2c461907233d4b90cae1267fdce864\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.655502 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.655464 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.755848 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.755770 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.784322 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.784283 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.789598 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:31.789003 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Aug 04 15:56:31.857327 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.857285 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:31.965076 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:31.965036 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:32.065545 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:32.065457 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:32.165725 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:32.165687 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:32.206312 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.206279 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 15:56:32.206777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.206448 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 15:56:32.266334 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:32.266305 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:32.290006 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.289967 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 15:51:32 +0000 UTC" deadline="2028-01-10 06:41:37.127114017 +0000 UTC" Aug 04 15:56:32.290006 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.290001 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12566h45m4.837117006s" Aug 04 15:56:32.359246 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:32.359191 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2c461907233d4b90cae1267fdce864.slice/crio-edc4d25dff27aa0a7ef1380ced2d7c7561477b7c7a4a4156db353197db771711 WatchSource:0}: Error finding container edc4d25dff27aa0a7ef1380ced2d7c7561477b7c7a4a4156db353197db771711: Status 404 returned error can't find the container with id edc4d25dff27aa0a7ef1380ced2d7c7561477b7c7a4a4156db353197db771711 Aug 04 15:56:32.364746 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.364725 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:56:32.366449 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:32.366425 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:32.466730 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:32.466685 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:32.541628 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.541596 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:32.567880 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:32.567854 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-14.ec2.internal\" not found" Aug 04 15:56:32.632898 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.632814 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:32.651225 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.651184 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:32.661536 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.661512 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" Aug 04 15:56:32.674682 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.674659 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 15:56:32.675762 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.675748 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" Aug 04 15:56:32.686830 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:32.686806 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 15:56:33.242589 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.242555 2571 apiserver.go:52] "Watching apiserver" Aug 04 15:56:33.252093 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.252060 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 15:56:33.253379 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.253351 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-swj4m","openshift-network-diagnostics/network-check-target-6xd6d","kube-system/konnectivity-agent-8f7lc","kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv","openshift-dns/node-resolver-vbw9c","openshift-image-registry/node-ca-7pxh8","openshift-multus/network-metrics-daemon-ltd4v","openshift-network-operator/iptables-alerter-xzsr9","openshift-ovn-kubernetes/ovnkube-node-dsn4r","kube-system/global-pull-secret-syncer-dknzx","openshift-cluster-node-tuning-operator/tuned-hfmld","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal","openshift-multus/multus-additional-cni-plugins-6zhbq"] Aug 04 15:56:33.257261 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.257237 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.258617 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.258566 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.258723 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.258662 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:33.260200 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.260176 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.260976 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.260953 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 15:56:33.261085 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.261068 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 15:56:33.261150 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.261091 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6hmmr\"" Aug 04 15:56:33.261150 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.261122 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 15:56:33.261286 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.261161 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 15:56:33.261508 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.261484 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 15:56:33.261621 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.261601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.262152 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.262057 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 15:56:33.262152 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.262138 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 15:56:33.263095 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.263075 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.264668 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.264510 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.264774 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.264714 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 15:56:33.265036 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.265018 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 15:56:33.265674 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.265654 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2lqcl\"" Aug 04 15:56:33.266717 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.266268 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 15:56:33.266717 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.266275 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 15:56:33.266717 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.266568 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 15:56:33.266717 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.266630 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.266717 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.266645 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 15:56:33.267010 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.266883 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 15:56:33.267067 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.267020 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 15:56:33.267832 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.267208 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 15:56:33.267832 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.267506 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m5r5t\"" Aug 04 15:56:33.267832 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.267624 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5gkdk\"" Aug 04 15:56:33.268003 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.267873 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 15:56:33.268003 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.267895 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 15:56:33.268322 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.268256 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 15:56:33.268322 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.268267 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zvgfv\"" Aug 04 15:56:33.268911 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.268894 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:33.269011 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.268989 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:33.270510 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.270478 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:33.270592 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.270565 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:33.270951 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.270930 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:56:33.271244 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.271206 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pcxr4\"" Aug 04 15:56:33.271244 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.271242 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 15:56:33.271443 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.271417 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 15:56:33.272271 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272252 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.272368 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.272368 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272329 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.272368 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272359 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03867d85-2528-4dfa-9618-8e8d14aeca1a-konnectivity-ca\") pod \"konnectivity-agent-8f7lc\" (UID: \"03867d85-2528-4dfa-9618-8e8d14aeca1a\") " pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.272531 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272392 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovnkube-script-lib\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272531 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272422 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/818a2360-3c98-4f8d-b540-d1c90b7d1113-dbus\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.272531 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272443 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.272531 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272462 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-cni-netd\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272531 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272481 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnw6j\" (UniqueName: \"kubernetes.io/projected/335a3459-a692-4ab6-a65f-88f3e4e8a53b-kube-api-access-fnw6j\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272531 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272500 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-cnibin\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.272531 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272520 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-log-socket\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272554 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-etc-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272571 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-os-release\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272599 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-registration-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-device-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272637 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-slash\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272657 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-cni-bin\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovnkube-config\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-system-cni-dir\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272724 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272758 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qldqw\" (UniqueName: \"kubernetes.io/projected/a32c5e48-78e8-4816-80c3-2d38d50a359f-kube-api-access-qldqw\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-socket-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.272885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272869 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-systemd-units\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272919 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-run-netns\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.272950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-var-lib-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273003 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-node-log\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273119 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk6ks\" (UniqueName: \"kubernetes.io/projected/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-kube-api-access-jk6ks\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273163 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-kubelet\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273188 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovn-node-metrics-cert\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/818a2360-3c98-4f8d-b540-d1c90b7d1113-kubelet-config\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03867d85-2528-4dfa-9618-8e8d14aeca1a-agent-certs\") pod \"konnectivity-agent-8f7lc\" (UID: \"03867d85-2528-4dfa-9618-8e8d14aeca1a\") " pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273353 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273391 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273417 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-sys-fs\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-systemd\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.273480 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273465 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-ovn\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.274093 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.274093 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273539 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-env-overrides\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.274093 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.273566 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.274093 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.274048 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.276127 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.276065 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.276290 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.276263 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 15:56:33.277211 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.277194 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ftp6n\"" Aug 04 15:56:33.282502 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.282479 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 15:56:33.282609 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.282523 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 15:56:33.282721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.282704 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 15:56:33.282793 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.282775 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s2tnd\"" Aug 04 15:56:33.282849 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.282792 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rxqrm\"" Aug 04 15:56:33.282894 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.282792 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:56:33.285244 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.285220 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 15:56:33.285330 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.285258 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 15:56:33.333343 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.333316 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j5mhd" Aug 04 15:56:33.345683 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.345657 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j5mhd" Aug 04 15:56:33.361544 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.361489 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" event={"ID":"2b2c461907233d4b90cae1267fdce864","Type":"ContainerStarted","Data":"edc4d25dff27aa0a7ef1380ced2d7c7561477b7c7a4a4156db353197db771711"} Aug 04 15:56:33.363392 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.363373 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 15:56:33.374020 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.373989 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-k8s-cni-cncf-io\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.374139 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374038 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-etc-kubernetes\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.374139 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-modprobe-d\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.374139 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374093 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-kubernetes\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.374139 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374121 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-systemd\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.374346 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374147 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strq5\" (UniqueName: \"kubernetes.io/projected/565e387c-e340-4c71-bf1e-873bfe4743a1-kube-api-access-strq5\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.374346 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03867d85-2528-4dfa-9618-8e8d14aeca1a-konnectivity-ca\") pod \"konnectivity-agent-8f7lc\" (UID: \"03867d85-2528-4dfa-9618-8e8d14aeca1a\") " pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.374346 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovnkube-script-lib\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374346 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374306 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.374506 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374423 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-cnibin\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.374506 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374445 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.374506 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374470 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-netns\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.374506 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374502 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2w6h\" (UniqueName: \"kubernetes.io/projected/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-kube-api-access-w2w6h\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:33.374684 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374528 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-var-lib-kubelet\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.374684 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-cni-netd\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374684 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw6j\" (UniqueName: \"kubernetes.io/projected/335a3459-a692-4ab6-a65f-88f3e4e8a53b-kube-api-access-fnw6j\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374684 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374641 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-log-socket\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374684 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374667 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-etc-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-device-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374718 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-os-release\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-kubelet\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374769 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374807 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-cni-netd\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-cni-bin\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374834 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-log-socket\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374846 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03867d85-2528-4dfa-9618-8e8d14aeca1a-konnectivity-ca\") pod \"konnectivity-agent-8f7lc\" (UID: \"03867d85-2528-4dfa-9618-8e8d14aeca1a\") " pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374858 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-cni-bin\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-system-cni-dir\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374807 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-etc-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374853 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-device-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374900 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.374924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-socket-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374961 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nr6b\" (UniqueName: \"kubernetes.io/projected/09b6e3bc-f0a6-4434-bc66-40e886747bf1-kube-api-access-7nr6b\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374903 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-system-cni-dir\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.374992 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-cni-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375064 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfdb\" (UniqueName: \"kubernetes.io/projected/145a5d49-25db-4674-838c-c2ce7612deb8-kube-api-access-wvfdb\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375093 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-socket-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375118 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-systemd-units\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375147 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-run-netns\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375201 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk6ks\" (UniqueName: \"kubernetes.io/projected/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-kube-api-access-jk6ks\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375231 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-hostroot\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375257 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-run-netns\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhtnp\" (UniqueName: \"kubernetes.io/projected/e97926c6-57de-4b89-9fdb-fe0d7310fb68-kube-api-access-bhtnp\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375319 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-systemd-units\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375319 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysctl-d\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375372 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375374 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysctl-conf\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.375721 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375408 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03867d85-2528-4dfa-9618-8e8d14aeca1a-agent-certs\") pod \"konnectivity-agent-8f7lc\" (UID: \"03867d85-2528-4dfa-9618-8e8d14aeca1a\") " pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375442 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375470 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375499 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375545 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba50246f-15ba-4589-9ae4-f38e8169ceec-hosts-file\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba50246f-15ba-4589-9ae4-f38e8169ceec-tmp-dir\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375597 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/565e387c-e340-4c71-bf1e-873bfe4743a1-tmp\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375624 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-ovn\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375657 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-env-overrides\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375669 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovnkube-script-lib\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375760 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-ovn\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375808 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375819 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375851 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-run\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375879 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-sys\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375911 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375938 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-socket-dir-parent\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.376646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375967 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-multus-certs\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.375994 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh267\" (UniqueName: \"kubernetes.io/projected/ba50246f-15ba-4589-9ae4-f38e8169ceec-kube-api-access-mh267\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/818a2360-3c98-4f8d-b540-d1c90b7d1113-dbus\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376032 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-env-overrides\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.376040 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376050 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-tuned\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-cnibin\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.376150 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret podName:818a2360-3c98-4f8d-b540-d1c90b7d1113 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:33.876104757 +0000 UTC m=+2.969705004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret") pod "global-pull-secret-syncer-dknzx" (UID: "818a2360-3c98-4f8d-b540-d1c90b7d1113") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376163 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-cnibin\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376174 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/145a5d49-25db-4674-838c-c2ce7612deb8-multus-daemon-config\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376198 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376221 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e97926c6-57de-4b89-9fdb-fe0d7310fb68-host\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376268 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysconfig\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376288 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/818a2360-3c98-4f8d-b540-d1c90b7d1113-dbus\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-host\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-os-release\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376431 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-registration-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.377507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376498 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-registration-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09b6e3bc-f0a6-4434-bc66-40e886747bf1-iptables-alerter-script\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376535 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-cni-bin\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376577 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a32c5e48-78e8-4816-80c3-2d38d50a359f-os-release\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376590 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-cni-multus\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-conf-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-lib-modules\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376687 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-slash\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376709 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovnkube-config\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376726 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qldqw\" (UniqueName: \"kubernetes.io/projected/a32c5e48-78e8-4816-80c3-2d38d50a359f-kube-api-access-qldqw\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-var-lib-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-node-log\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376785 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-slash\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376828 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/145a5d49-25db-4674-838c-c2ce7612deb8-cni-binary-copy\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376852 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-var-lib-openvswitch\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.378372 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376853 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e97926c6-57de-4b89-9fdb-fe0d7310fb68-serviceca\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376887 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-kubelet\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.376976 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-node-log\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377019 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-kubelet\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377060 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovn-node-metrics-cert\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377120 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/818a2360-3c98-4f8d-b540-d1c90b7d1113-kubelet-config\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377175 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-sys-fs\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-systemd\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377258 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377270 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/818a2360-3c98-4f8d-b540-d1c90b7d1113-kubelet-config\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377302 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovnkube-config\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377315 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-run-systemd\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377313 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377364 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-sys-fs\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377378 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-system-cni-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377388 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/335a3459-a692-4ab6-a65f-88f3e4e8a53b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377391 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.379710 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377421 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.379710 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377449 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09b6e3bc-f0a6-4434-bc66-40e886747bf1-host-slash\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.379710 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.377913 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32c5e48-78e8-4816-80c3-2d38d50a359f-cni-binary-copy\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.379903 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.379879 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/335a3459-a692-4ab6-a65f-88f3e4e8a53b-ovn-node-metrics-cert\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.379997 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.379980 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03867d85-2528-4dfa-9618-8e8d14aeca1a-agent-certs\") pod \"konnectivity-agent-8f7lc\" (UID: \"03867d85-2528-4dfa-9618-8e8d14aeca1a\") " pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.393494 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.393468 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qldqw\" (UniqueName: \"kubernetes.io/projected/a32c5e48-78e8-4816-80c3-2d38d50a359f-kube-api-access-qldqw\") pod \"multus-additional-cni-plugins-6zhbq\" (UID: \"a32c5e48-78e8-4816-80c3-2d38d50a359f\") " pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.393608 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.393590 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw6j\" (UniqueName: \"kubernetes.io/projected/335a3459-a692-4ab6-a65f-88f3e4e8a53b-kube-api-access-fnw6j\") pod \"ovnkube-node-dsn4r\" (UID: \"335a3459-a692-4ab6-a65f-88f3e4e8a53b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.394037 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.394012 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk6ks\" (UniqueName: \"kubernetes.io/projected/772659eb-e9a4-4daa-974a-4f2fdf3c91b4-kube-api-access-jk6ks\") pod \"aws-ebs-csi-driver-node-8txtv\" (UID: \"772659eb-e9a4-4daa-974a-4f2fdf3c91b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.477791 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477751 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/145a5d49-25db-4674-838c-c2ce7612deb8-multus-daemon-config\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.477791 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e97926c6-57de-4b89-9fdb-fe0d7310fb68-host\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477817 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysconfig\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-host\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09b6e3bc-f0a6-4434-bc66-40e886747bf1-iptables-alerter-script\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477899 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-cni-bin\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477921 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e97926c6-57de-4b89-9fdb-fe0d7310fb68-host\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-cni-multus\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.477983 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-cni-multus\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478000 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-conf-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.478076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478044 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-lib-modules\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.478493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478433 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysconfig\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.479519 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478601 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-lib-modules\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.479519 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478833 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:33.479519 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478880 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-conf-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.479519 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478887 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/145a5d49-25db-4674-838c-c2ce7612deb8-cni-binary-copy\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.479519 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e97926c6-57de-4b89-9fdb-fe0d7310fb68-serviceca\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.479519 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.478974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-system-cni-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.479914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09b6e3bc-f0a6-4434-bc66-40e886747bf1-host-slash\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.479914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-k8s-cni-cncf-io\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.479914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-etc-kubernetes\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.479914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479832 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-modprobe-d\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.479914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-kubernetes\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.479914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479907 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-systemd\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.480218 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strq5\" (UniqueName: \"kubernetes.io/projected/565e387c-e340-4c71-bf1e-873bfe4743a1-kube-api-access-strq5\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.480218 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479980 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-cnibin\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480218 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480017 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-netns\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480218 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480169 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e97926c6-57de-4b89-9fdb-fe0d7310fb68-serviceca\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.480218 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480180 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2w6h\" (UniqueName: \"kubernetes.io/projected/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-kube-api-access-w2w6h\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479383 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-cni-bin\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479250 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09b6e3bc-f0a6-4434-bc66-40e886747bf1-iptables-alerter-script\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-var-lib-kubelet\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480308 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-k8s-cni-cncf-io\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480315 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-etc-kubernetes\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-os-release\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480398 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-kubernetes\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479477 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/145a5d49-25db-4674-838c-c2ce7612deb8-cni-binary-copy\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480493 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-kubelet\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480499 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480519 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-systemd\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.479585 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480545 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nr6b\" (UniqueName: \"kubernetes.io/projected/09b6e3bc-f0a6-4434-bc66-40e886747bf1-kube-api-access-7nr6b\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480581 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-cni-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479042 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-system-cni-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480620 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfdb\" (UniqueName: \"kubernetes.io/projected/145a5d49-25db-4674-838c-c2ce7612deb8-kube-api-access-wvfdb\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480683 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-hostroot\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.480753 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:56:33.98071372 +0000 UTC m=+3.074313963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480763 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-hostroot\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhtnp\" (UniqueName: \"kubernetes.io/projected/e97926c6-57de-4b89-9fdb-fe0d7310fb68-kube-api-access-bhtnp\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480815 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysctl-d\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480822 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09b6e3bc-f0a6-4434-bc66-40e886747bf1-host-slash\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479054 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/145a5d49-25db-4674-838c-c2ce7612deb8-multus-daemon-config\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.479339 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-host\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.480934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.480884 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysctl-conf\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481019 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-cnibin\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481068 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysctl-conf\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481126 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-netns\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481197 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-sysctl-d\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba50246f-15ba-4589-9ae4-f38e8169ceec-hosts-file\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba50246f-15ba-4589-9ae4-f38e8169ceec-tmp-dir\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481333 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-var-lib-kubelet\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481327 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/565e387c-e340-4c71-bf1e-873bfe4743a1-tmp\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481421 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-var-lib-kubelet\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481461 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-run\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481486 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-sys\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-socket-dir-parent\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481580 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-multus-certs\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481608 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh267\" (UniqueName: \"kubernetes.io/projected/ba50246f-15ba-4589-9ae4-f38e8169ceec-kube-api-access-mh267\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.481807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.481644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-tuned\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.482611 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.482057 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-cni-dir\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.482611 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.482140 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba50246f-15ba-4589-9ae4-f38e8169ceec-hosts-file\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.482611 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.482557 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba50246f-15ba-4589-9ae4-f38e8169ceec-tmp-dir\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.482772 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.482745 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-sys\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.483168 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.482900 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-modprobe-d\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.483168 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.482990 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-os-release\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.483168 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.483045 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/565e387c-e340-4c71-bf1e-873bfe4743a1-run\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.483168 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.483115 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-host-run-multus-certs\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.483575 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.483550 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/145a5d49-25db-4674-838c-c2ce7612deb8-multus-socket-dir-parent\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.484640 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.484596 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/565e387c-e340-4c71-bf1e-873bfe4743a1-tmp\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.484640 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.484626 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/565e387c-e340-4c71-bf1e-873bfe4743a1-etc-tuned\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.493944 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.493893 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:33.493944 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.493918 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:33.493944 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.493930 2571 projected.go:196] Error preparing data for projected volume kube-api-access-d8m4k for pod openshift-network-diagnostics/network-check-target-6xd6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:33.494134 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.493991 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k podName:77e34eb5-f2d6-4f4b-bb61-78fed95fa688 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:33.993977389 +0000 UTC m=+3.087577616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d8m4k" (UniqueName: "kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k") pod "network-check-target-6xd6d" (UID: "77e34eb5-f2d6-4f4b-bb61-78fed95fa688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:33.494763 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.494736 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-strq5\" (UniqueName: \"kubernetes.io/projected/565e387c-e340-4c71-bf1e-873bfe4743a1-kube-api-access-strq5\") pod \"tuned-hfmld\" (UID: \"565e387c-e340-4c71-bf1e-873bfe4743a1\") " pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.497537 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.497511 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nr6b\" (UniqueName: \"kubernetes.io/projected/09b6e3bc-f0a6-4434-bc66-40e886747bf1-kube-api-access-7nr6b\") pod \"iptables-alerter-xzsr9\" (UID: \"09b6e3bc-f0a6-4434-bc66-40e886747bf1\") " pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.498562 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.498531 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh267\" (UniqueName: \"kubernetes.io/projected/ba50246f-15ba-4589-9ae4-f38e8169ceec-kube-api-access-mh267\") pod \"node-resolver-vbw9c\" (UID: \"ba50246f-15ba-4589-9ae4-f38e8169ceec\") " pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.498666 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.498607 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfdb\" (UniqueName: \"kubernetes.io/projected/145a5d49-25db-4674-838c-c2ce7612deb8-kube-api-access-wvfdb\") pod \"multus-swj4m\" (UID: \"145a5d49-25db-4674-838c-c2ce7612deb8\") " pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.498771 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.498753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhtnp\" (UniqueName: \"kubernetes.io/projected/e97926c6-57de-4b89-9fdb-fe0d7310fb68-kube-api-access-bhtnp\") pod \"node-ca-7pxh8\" (UID: \"e97926c6-57de-4b89-9fdb-fe0d7310fb68\") " pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.499490 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.499466 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2w6h\" (UniqueName: \"kubernetes.io/projected/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-kube-api-access-w2w6h\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:33.572279 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.572243 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:56:33.580888 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.580863 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:33.588788 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.588763 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" Aug 04 15:56:33.597418 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.597399 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" Aug 04 15:56:33.605182 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.605161 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vbw9c" Aug 04 15:56:33.611923 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.611894 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xzsr9" Aug 04 15:56:33.618402 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.618384 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-swj4m" Aug 04 15:56:33.624020 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.624002 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7pxh8" Aug 04 15:56:33.628914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.628891 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hfmld" Aug 04 15:56:33.689004 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.688980 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:56:33.884058 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.883972 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:33.884202 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.884124 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:33.884202 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.884189 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret podName:818a2360-3c98-4f8d-b540-d1c90b7d1113 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:34.884174647 +0000 UTC m=+3.977774878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret") pod "global-pull-secret-syncer-dknzx" (UID: "818a2360-3c98-4f8d-b540-d1c90b7d1113") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:33.984572 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:33.984538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:33.984793 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.984665 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:33.984793 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:33.984727 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:56:34.98471042 +0000 UTC m=+4.078310649 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:34.016710 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.016674 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod145a5d49_25db_4674_838c_c2ce7612deb8.slice/crio-edb95510830cc88276935651e79e0746eb61739141498a80381c799d2acdfa24 WatchSource:0}: Error finding container edb95510830cc88276935651e79e0746eb61739141498a80381c799d2acdfa24: Status 404 returned error can't find the container with id edb95510830cc88276935651e79e0746eb61739141498a80381c799d2acdfa24 Aug 04 15:56:34.017638 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.017609 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03867d85_2528_4dfa_9618_8e8d14aeca1a.slice/crio-1ea5d5bcd58ccd39762958dbe3d15181363c044760f101f5259b6e731bc7ffbf WatchSource:0}: Error finding container 1ea5d5bcd58ccd39762958dbe3d15181363c044760f101f5259b6e731bc7ffbf: Status 404 returned error can't find the container with id 1ea5d5bcd58ccd39762958dbe3d15181363c044760f101f5259b6e731bc7ffbf Aug 04 15:56:34.022196 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.022092 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32c5e48_78e8_4816_80c3_2d38d50a359f.slice/crio-1c41aa1e208ed5838794df0955d23b4dfab576fba8a53c3bb68937b3f3a9684b WatchSource:0}: Error finding container 1c41aa1e208ed5838794df0955d23b4dfab576fba8a53c3bb68937b3f3a9684b: Status 404 returned error can't find the container with id 1c41aa1e208ed5838794df0955d23b4dfab576fba8a53c3bb68937b3f3a9684b Aug 04 15:56:34.022654 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.022617 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba50246f_15ba_4589_9ae4_f38e8169ceec.slice/crio-19ade9df93970ce21cb1623c618a936c6cdcefe7af9d0b20e8918126d6dcf51c WatchSource:0}: Error finding container 19ade9df93970ce21cb1623c618a936c6cdcefe7af9d0b20e8918126d6dcf51c: Status 404 returned error can't find the container with id 19ade9df93970ce21cb1623c618a936c6cdcefe7af9d0b20e8918126d6dcf51c Aug 04 15:56:34.024329 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.024307 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97926c6_57de_4b89_9fdb_fe0d7310fb68.slice/crio-b294f0571e2d672d2f7b9e889bfa30245726f9784c9a848bf104b698ab4d45a0 WatchSource:0}: Error finding container b294f0571e2d672d2f7b9e889bfa30245726f9784c9a848bf104b698ab4d45a0: Status 404 returned error can't find the container with id b294f0571e2d672d2f7b9e889bfa30245726f9784c9a848bf104b698ab4d45a0 Aug 04 15:56:34.025347 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.025324 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b6e3bc_f0a6_4434_bc66_40e886747bf1.slice/crio-0fa9b540a397ca0158f1537c807d60b216e15e1fdec600e260a66d633134596e WatchSource:0}: Error finding container 0fa9b540a397ca0158f1537c807d60b216e15e1fdec600e260a66d633134596e: Status 404 returned error can't find the container with id 0fa9b540a397ca0158f1537c807d60b216e15e1fdec600e260a66d633134596e Aug 04 15:56:34.027344 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.027318 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772659eb_e9a4_4daa_974a_4f2fdf3c91b4.slice/crio-d5acf99e3d2479209cfcbdb89e00b39753c0070e13da7f8c3f9ca63b0ff4fc4c WatchSource:0}: Error finding container d5acf99e3d2479209cfcbdb89e00b39753c0070e13da7f8c3f9ca63b0ff4fc4c: Status 404 returned error can't find the container with id d5acf99e3d2479209cfcbdb89e00b39753c0070e13da7f8c3f9ca63b0ff4fc4c Aug 04 15:56:34.028103 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.028064 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335a3459_a692_4ab6_a65f_88f3e4e8a53b.slice/crio-1241f4d7a3134aacf966332de8805b521e7cfcd449ad5cc3cf7c86d8fe8dd343 WatchSource:0}: Error finding container 1241f4d7a3134aacf966332de8805b521e7cfcd449ad5cc3cf7c86d8fe8dd343: Status 404 returned error can't find the container with id 1241f4d7a3134aacf966332de8805b521e7cfcd449ad5cc3cf7c86d8fe8dd343 Aug 04 15:56:34.029428 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.029054 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565e387c_e340_4c71_bf1e_873bfe4743a1.slice/crio-995ce3d565343f4ec461acd68eedd044b843398140a5997d3cea3db39f55f208 WatchSource:0}: Error finding container 995ce3d565343f4ec461acd68eedd044b843398140a5997d3cea3db39f55f208: Status 404 returned error can't find the container with id 995ce3d565343f4ec461acd68eedd044b843398140a5997d3cea3db39f55f208 Aug 04 15:56:34.085107 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.085077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:34.085364 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.085281 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:34.085364 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.085309 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:34.085364 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.085325 2571 projected.go:196] Error preparing data for projected volume kube-api-access-d8m4k for pod openshift-network-diagnostics/network-check-target-6xd6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:34.085509 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.085398 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k podName:77e34eb5-f2d6-4f4b-bb61-78fed95fa688 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:35.085383423 +0000 UTC m=+4.178983651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8m4k" (UniqueName: "kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k") pod "network-check-target-6xd6d" (UID: "77e34eb5-f2d6-4f4b-bb61-78fed95fa688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:34.232335 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:56:34.232282 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11916bc9c437c84f693e9d6334241cc.slice/crio-4eb8f6b2ae5ec134859b00be481fa8b3081638fcb483648b4d85e9a4b8258c8e WatchSource:0}: Error finding container 4eb8f6b2ae5ec134859b00be481fa8b3081638fcb483648b4d85e9a4b8258c8e: Status 404 returned error can't find the container with id 4eb8f6b2ae5ec134859b00be481fa8b3081638fcb483648b4d85e9a4b8258c8e Aug 04 15:56:34.349499 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.349457 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:51:33 +0000 UTC" deadline="2028-02-03 22:47:04.547977209 +0000 UTC" Aug 04 15:56:34.349499 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.349491 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13158h50m30.198489357s" Aug 04 15:56:34.356810 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.356775 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:34.356919 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.356909 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:34.356960 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.356937 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:34.357014 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.356973 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:34.364740 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.364715 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8f7lc" event={"ID":"03867d85-2528-4dfa-9618-8e8d14aeca1a","Type":"ContainerStarted","Data":"1ea5d5bcd58ccd39762958dbe3d15181363c044760f101f5259b6e731bc7ffbf"} Aug 04 15:56:34.366062 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.366027 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swj4m" event={"ID":"145a5d49-25db-4674-838c-c2ce7612deb8","Type":"ContainerStarted","Data":"edb95510830cc88276935651e79e0746eb61739141498a80381c799d2acdfa24"} Aug 04 15:56:34.367254 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.367230 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" event={"ID":"e11916bc9c437c84f693e9d6334241cc","Type":"ContainerStarted","Data":"4eb8f6b2ae5ec134859b00be481fa8b3081638fcb483648b4d85e9a4b8258c8e"} Aug 04 15:56:34.368326 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.368305 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfmld" event={"ID":"565e387c-e340-4c71-bf1e-873bfe4743a1","Type":"ContainerStarted","Data":"995ce3d565343f4ec461acd68eedd044b843398140a5997d3cea3db39f55f208"} Aug 04 15:56:34.369880 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.369860 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"1241f4d7a3134aacf966332de8805b521e7cfcd449ad5cc3cf7c86d8fe8dd343"} Aug 04 15:56:34.371114 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.371096 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" event={"ID":"772659eb-e9a4-4daa-974a-4f2fdf3c91b4","Type":"ContainerStarted","Data":"d5acf99e3d2479209cfcbdb89e00b39753c0070e13da7f8c3f9ca63b0ff4fc4c"} Aug 04 15:56:34.372222 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.372181 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzsr9" event={"ID":"09b6e3bc-f0a6-4434-bc66-40e886747bf1","Type":"ContainerStarted","Data":"0fa9b540a397ca0158f1537c807d60b216e15e1fdec600e260a66d633134596e"} Aug 04 15:56:34.373312 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.373293 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pxh8" event={"ID":"e97926c6-57de-4b89-9fdb-fe0d7310fb68","Type":"ContainerStarted","Data":"b294f0571e2d672d2f7b9e889bfa30245726f9784c9a848bf104b698ab4d45a0"} Aug 04 15:56:34.374416 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.374398 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerStarted","Data":"1c41aa1e208ed5838794df0955d23b4dfab576fba8a53c3bb68937b3f3a9684b"} Aug 04 15:56:34.375457 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.375435 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vbw9c" event={"ID":"ba50246f-15ba-4589-9ae4-f38e8169ceec","Type":"ContainerStarted","Data":"19ade9df93970ce21cb1623c618a936c6cdcefe7af9d0b20e8918126d6dcf51c"} Aug 04 15:56:34.377161 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.377143 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" event={"ID":"2b2c461907233d4b90cae1267fdce864","Type":"ContainerStarted","Data":"904fc0b35631ae74ad0737bd3a190deda47ae5e7184e833efdb4152a075c845b"} Aug 04 15:56:34.397116 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.397057 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-14.ec2.internal" podStartSLOduration=1.397042374 podStartE2EDuration="1.397042374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:34.396955726 +0000 UTC m=+3.490555978" watchObservedRunningTime="2026-08-04 15:56:34.397042374 +0000 UTC m=+3.490642625" Aug 04 15:56:34.892959 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.892368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:34.892959 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.892527 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:34.892959 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.892596 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret podName:818a2360-3c98-4f8d-b540-d1c90b7d1113 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:36.892575446 +0000 UTC m=+5.986175673 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret") pod "global-pull-secret-syncer-dknzx" (UID: "818a2360-3c98-4f8d-b540-d1c90b7d1113") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:34.993772 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:34.993730 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:34.993965 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.993950 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:34.994032 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:34.994018 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:56:36.99399893 +0000 UTC m=+6.087599164 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:35.094717 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:35.094671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:35.094897 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:35.094861 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:35.094897 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:35.094883 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:35.094897 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:35.094897 2571 projected.go:196] Error preparing data for projected volume kube-api-access-d8m4k for pod openshift-network-diagnostics/network-check-target-6xd6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:35.095070 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:35.094967 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k podName:77e34eb5-f2d6-4f4b-bb61-78fed95fa688 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:37.094945427 +0000 UTC m=+6.188545659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8m4k" (UniqueName: "kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k") pod "network-check-target-6xd6d" (UID: "77e34eb5-f2d6-4f4b-bb61-78fed95fa688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:35.349933 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:35.349766 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:51:33 +0000 UTC" deadline="2028-04-16 17:44:14.842417022 +0000 UTC" Aug 04 15:56:35.349933 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:35.349842 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14905h47m39.492579521s" Aug 04 15:56:35.359967 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:35.359400 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:35.359967 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:35.359545 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:36.358245 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:36.357536 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:36.358245 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:36.357679 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:36.358245 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:36.358104 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:36.358245 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:36.358197 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:36.912431 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:36.911489 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:36.912431 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:36.911940 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:36.912431 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:36.912011 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret podName:818a2360-3c98-4f8d-b540-d1c90b7d1113 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:40.911991003 +0000 UTC m=+10.005591234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret") pod "global-pull-secret-syncer-dknzx" (UID: "818a2360-3c98-4f8d-b540-d1c90b7d1113") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:37.013418 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:37.012702 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:37.013418 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:37.012931 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:37.013418 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:37.013000 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:56:41.012979653 +0000 UTC m=+10.106579884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:37.113401 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:37.113346 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:37.113600 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:37.113541 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:37.113600 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:37.113576 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:37.113600 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:37.113592 2571 projected.go:196] Error preparing data for projected volume kube-api-access-d8m4k for pod openshift-network-diagnostics/network-check-target-6xd6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:37.113793 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:37.113663 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k podName:77e34eb5-f2d6-4f4b-bb61-78fed95fa688 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:41.113643058 +0000 UTC m=+10.207243302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8m4k" (UniqueName: "kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k") pod "network-check-target-6xd6d" (UID: "77e34eb5-f2d6-4f4b-bb61-78fed95fa688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:37.360700 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:37.360072 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:37.360700 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:37.360227 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:38.357064 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:38.357026 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:38.357318 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:38.357187 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:38.357318 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:38.357205 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:38.357466 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:38.357348 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:39.357243 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:39.357114 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:39.358162 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:39.357797 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:40.357434 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:40.357395 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:40.357969 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:40.357557 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:40.359271 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:40.358403 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:40.359271 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:40.358557 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:40.949221 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:40.949160 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:40.949406 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:40.949361 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:40.949468 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:40.949436 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret podName:818a2360-3c98-4f8d-b540-d1c90b7d1113 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:48.949415162 +0000 UTC m=+18.043015414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret") pod "global-pull-secret-syncer-dknzx" (UID: "818a2360-3c98-4f8d-b540-d1c90b7d1113") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:41.050076 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:41.049995 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:41.050266 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:41.050112 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:41.050266 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:41.050179 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:56:49.050160942 +0000 UTC m=+18.143761184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:41.150995 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:41.150953 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:41.151168 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:41.151123 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:41.151168 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:41.151149 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:41.151168 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:41.151163 2571 projected.go:196] Error preparing data for projected volume kube-api-access-d8m4k for pod openshift-network-diagnostics/network-check-target-6xd6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:41.151318 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:41.151250 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k podName:77e34eb5-f2d6-4f4b-bb61-78fed95fa688 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:49.151230276 +0000 UTC m=+18.244830507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8m4k" (UniqueName: "kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k") pod "network-check-target-6xd6d" (UID: "77e34eb5-f2d6-4f4b-bb61-78fed95fa688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:41.358134 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:41.358104 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:41.358593 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:41.358249 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:42.357571 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:42.357537 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:42.357740 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:42.357588 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:42.357740 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:42.357663 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:42.357859 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:42.357745 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:43.357151 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:43.357109 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:43.357616 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:43.357273 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:44.357353 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:44.357314 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:44.358063 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:44.357455 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:44.358063 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:44.357543 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:44.358063 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:44.357662 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:45.357021 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:45.356985 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:45.357223 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:45.357134 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:46.357027 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:46.356987 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:46.357488 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:46.357033 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:46.357488 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:46.357122 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:46.357488 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:46.357241 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:47.357540 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:47.357510 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:47.357948 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:47.357642 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:48.357389 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:48.357349 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:48.357389 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:48.357375 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:48.357961 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:48.357482 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:48.357961 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:48.357615 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:49.011842 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:49.011803 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:49.012039 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.011953 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:49.012039 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.012031 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret podName:818a2360-3c98-4f8d-b540-d1c90b7d1113 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:05.012011654 +0000 UTC m=+34.105611898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret") pod "global-pull-secret-syncer-dknzx" (UID: "818a2360-3c98-4f8d-b540-d1c90b7d1113") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:56:49.113121 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:49.113072 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:49.113310 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.113236 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:49.113384 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.113313 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:57:05.113291024 +0000 UTC m=+34.206891250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:56:49.213730 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:49.213695 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:49.213956 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.213876 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:56:49.213956 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.213903 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:56:49.213956 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.213916 2571 projected.go:196] Error preparing data for projected volume kube-api-access-d8m4k for pod openshift-network-diagnostics/network-check-target-6xd6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:49.214107 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.213970 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k podName:77e34eb5-f2d6-4f4b-bb61-78fed95fa688 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:05.213952159 +0000 UTC m=+34.307552386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8m4k" (UniqueName: "kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k") pod "network-check-target-6xd6d" (UID: "77e34eb5-f2d6-4f4b-bb61-78fed95fa688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:56:49.357247 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:49.357197 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:49.357485 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:49.357336 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:50.356901 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:50.356811 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:50.356901 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:50.356858 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:50.357343 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:50.356948 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:50.357343 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:50.357069 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:51.357699 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:51.357657 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:51.358111 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:51.357816 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:52.356933 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:52.356896 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:52.357127 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:52.356899 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:52.357127 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:52.357012 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:52.357266 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:52.357130 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:53.357230 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:53.357184 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:53.357721 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:53.357322 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:54.357787 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.357612 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:54.358374 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.357659 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:54.358374 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:54.357867 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:54.358374 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:54.357958 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:54.423951 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.423916 2571 generic.go:356] "Generic (PLEG): container finished" podID="e11916bc9c437c84f693e9d6334241cc" containerID="50a380c1f10319783eebf0fbed5ee4e57b58e144e1840cd04f0f7f3c7e8d5974" exitCode=0 Aug 04 15:56:54.424106 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.424001 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" event={"ID":"e11916bc9c437c84f693e9d6334241cc","Type":"ContainerDied","Data":"50a380c1f10319783eebf0fbed5ee4e57b58e144e1840cd04f0f7f3c7e8d5974"} Aug 04 15:56:54.425489 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.425465 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hfmld" event={"ID":"565e387c-e340-4c71-bf1e-873bfe4743a1","Type":"ContainerStarted","Data":"1ba41ce430a9159da989326917d421762a1dba4759e9fc962d595bb5cc939730"} Aug 04 15:56:54.428223 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.428183 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"f7f1243996fe984b0a631118fcbe60dc204a1bf2089b4079fb16950a69ba9481"} Aug 04 15:56:54.428335 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.428284 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"b1d8fb41bb08bed05535252ea6ce694999e8ec7a044a752e079c25f45ee108ef"} Aug 04 15:56:54.428399 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.428338 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"f9f10b1b12c17b99ed98656a2f4bf87d9d36f2c3eb743582942c959aac125628"} Aug 04 15:56:54.428399 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.428390 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"757b7a059debe93880c893fd2516641de8265c1fe85dfd7b5c6a5676b4d67d3a"} Aug 04 15:56:54.428501 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.428442 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"a4602b230017e07772a740eb6bcf736510377f89258be570abbfc30901e379d9"} Aug 04 15:56:54.428501 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.428489 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"e81cf1b24595154f01a9394020b364545426a4612fe2fdcb1703586195e97a37"} Aug 04 15:56:54.429518 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.429495 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" event={"ID":"772659eb-e9a4-4daa-974a-4f2fdf3c91b4","Type":"ContainerStarted","Data":"1e680160dce50216626c1e5fe17cdd30dfceb6204e18f44302e8b2d0ac539221"} Aug 04 15:56:54.430749 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.430725 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7pxh8" event={"ID":"e97926c6-57de-4b89-9fdb-fe0d7310fb68","Type":"ContainerStarted","Data":"8816bae7f7fc531dad8c0ba18c5da53faffc3b91028061e983800f5b53c674b5"} Aug 04 15:56:54.432002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.431977 2571 generic.go:356] "Generic (PLEG): container finished" podID="a32c5e48-78e8-4816-80c3-2d38d50a359f" containerID="83c043480d58bf09ad4474daaa5f83a18ee43c1cc029f36f4554ecbf9620ee50" exitCode=0 Aug 04 15:56:54.432096 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.432002 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerDied","Data":"83c043480d58bf09ad4474daaa5f83a18ee43c1cc029f36f4554ecbf9620ee50"} Aug 04 15:56:54.433356 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.433298 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vbw9c" event={"ID":"ba50246f-15ba-4589-9ae4-f38e8169ceec","Type":"ContainerStarted","Data":"480ffbd417ae3afcbac47d5a0715e2a432f22d8ebe58b7ceeda57247ae168d47"} Aug 04 15:56:54.434647 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.434579 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8f7lc" event={"ID":"03867d85-2528-4dfa-9618-8e8d14aeca1a","Type":"ContainerStarted","Data":"59fad9a2c6b3feb83a8c14f9a60159e7dc811d48e4b747ee4221b1c13230b9f4"} Aug 04 15:56:54.435994 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.435977 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-swj4m" event={"ID":"145a5d49-25db-4674-838c-c2ce7612deb8","Type":"ContainerStarted","Data":"5339bb9d8fd74e71e288c19219d70b6e202ec67ce82012301585f8bd0ee2ea5e"} Aug 04 15:56:54.464576 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.464534 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8f7lc" podStartSLOduration=2.768619502 podStartE2EDuration="22.464519651s" totalImagesPullingTime="19.695900149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:54.463855565 +0000 UTC m=+23.557455813" watchObservedRunningTime="2026-08-04 15:56:54.464519651 +0000 UTC m=+23.558119898" Aug 04 15:56:54.484378 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.484301 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-swj4m" podStartSLOduration=2.730307285 podStartE2EDuration="22.484288825s" totalImagesPullingTime="19.75398154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:54.483708729 +0000 UTC m=+23.577308978" watchObservedRunningTime="2026-08-04 15:56:54.484288825 +0000 UTC m=+23.577889073" Aug 04 15:56:54.503338 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.503298 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7pxh8" podStartSLOduration=2.815219088 podStartE2EDuration="22.503287091s" totalImagesPullingTime="19.688068003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:54.503154063 +0000 UTC m=+23.596754311" watchObservedRunningTime="2026-08-04 15:56:54.503287091 +0000 UTC m=+23.596887339" Aug 04 15:56:54.546377 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.546137 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hfmld" podStartSLOduration=2.86324824 podStartE2EDuration="22.546121957s" totalImagesPullingTime="19.682873717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:54.522298038 +0000 UTC m=+23.615898286" watchObservedRunningTime="2026-08-04 15:56:54.546121957 +0000 UTC m=+23.639722206" Aug 04 15:56:54.568784 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.568740 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vbw9c" podStartSLOduration=2.865956089 podStartE2EDuration="22.568724864s" totalImagesPullingTime="19.702768775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:54.568518282 +0000 UTC m=+23.662118542" watchObservedRunningTime="2026-08-04 15:56:54.568724864 +0000 UTC m=+23.662325112" Aug 04 15:56:54.640432 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:54.640389 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:55.356722 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.356685 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:55.361351 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:55.360656 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:55.440803 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.440767 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" event={"ID":"e11916bc9c437c84f693e9d6334241cc","Type":"ContainerStarted","Data":"88e240a57de78a6b123e6d55dbd59b78d36a03780474c46e89b4b9040489dec6"} Aug 04 15:56:55.442920 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.442854 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xzsr9" event={"ID":"09b6e3bc-f0a6-4434-bc66-40e886747bf1","Type":"ContainerStarted","Data":"d683c736d95e03a977821b100e0c1b7d6caf756c5a5c2bbd4cd08e6a924f6064"} Aug 04 15:56:55.457995 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.457950 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-14.ec2.internal" podStartSLOduration=22.457934257 podStartE2EDuration="22.457934257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:55.457243793 +0000 UTC m=+24.550844042" watchObservedRunningTime="2026-08-04 15:56:55.457934257 +0000 UTC m=+24.551534505" Aug 04 15:56:55.474240 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.474157 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xzsr9" podStartSLOduration=3.781432133 podStartE2EDuration="23.474131975s" totalImagesPullingTime="19.692699842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:55.473196213 +0000 UTC m=+24.566796462" watchObservedRunningTime="2026-08-04 15:56:55.474131975 +0000 UTC m=+24.567732224" Aug 04 15:56:55.505101 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.505073 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 15:56:55.669882 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.669791 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:55.670501 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:55.670481 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:56.342124 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.342003 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T15:56:55.505092305Z","UUID":"8e4c296f-abfe-463a-b985-8751a52b9025","Handler":null,"Name":"","Endpoint":""} Aug 04 15:56:56.343843 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.343817 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 15:56:56.343981 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.343859 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 15:56:56.356714 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.356675 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:56.356945 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:56.356817 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:56.356945 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.356675 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:56.356945 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:56.356911 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:56.447145 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.447114 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" event={"ID":"772659eb-e9a4-4daa-974a-4f2fdf3c91b4","Type":"ContainerStarted","Data":"5acdfb6c6ee763bb08848e11417df530ccd54938264b78db5089dbb60728d262"} Aug 04 15:56:56.447589 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.447422 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:56.448060 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.448043 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8f7lc" Aug 04 15:56:56.890621 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.890530 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:56.989325 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:56.989290 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:57.357533 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:57.357495 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:57.357728 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:57.357637 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:56:57.454818 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:57.454778 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"a81176d4c339eceff9c41bb5b535af03663d414f0ea9f6c494c3ce1fb91496d8"} Aug 04 15:56:57.457275 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:57.457239 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" event={"ID":"772659eb-e9a4-4daa-974a-4f2fdf3c91b4","Type":"ContainerStarted","Data":"788f736c76b01d589ea658bb1783eb81da1bae5d425491403cbd284c754ff047"} Aug 04 15:56:57.479255 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:57.479185 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8txtv" podStartSLOduration=3.418061195 podStartE2EDuration="25.479166197s" totalImagesPullingTime="22.061105002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:57.478935515 +0000 UTC m=+26.572535764" watchObservedRunningTime="2026-08-04 15:56:57.479166197 +0000 UTC m=+26.572766473" Aug 04 15:56:58.357178 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:58.357141 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:56:58.357178 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:58.357158 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:56:58.357444 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:58.357269 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:56:58.357444 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:58.357346 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:56:59.357795 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:56:59.357761 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:56:59.358280 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:56:59.357886 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:57:00.357741 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.357512 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:00.357883 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.357512 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:00.357883 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:00.357766 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:57:00.357883 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:00.357838 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:57:00.468646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.468608 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" event={"ID":"335a3459-a692-4ab6-a65f-88f3e4e8a53b","Type":"ContainerStarted","Data":"f4b587d8347a6644cb2675417217feeefb642d5bd547a11158a229e409c26ce5"} Aug 04 15:57:00.468948 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.468928 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:57:00.470556 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.470531 2571 generic.go:356] "Generic (PLEG): container finished" podID="a32c5e48-78e8-4816-80c3-2d38d50a359f" containerID="d60b27ca8301022314af319693d4f60b905ab8328175bcbcbefb2ddf66ef9273" exitCode=0 Aug 04 15:57:00.470651 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.470574 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerDied","Data":"d60b27ca8301022314af319693d4f60b905ab8328175bcbcbefb2ddf66ef9273"} Aug 04 15:57:00.484952 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.484930 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:57:00.501283 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:00.501241 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" podStartSLOduration=8.764340593 podStartE2EDuration="28.501226128s" totalImagesPullingTime="19.736885535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:00.500802465 +0000 UTC m=+29.594402713" watchObservedRunningTime="2026-08-04 15:57:00.501226128 +0000 UTC m=+29.594826375" Aug 04 15:57:01.357967 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.357934 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:01.358449 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:01.358066 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:57:01.473659 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.473467 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:57:01.473805 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.473715 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:57:01.488519 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.488486 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:57:01.865393 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.865353 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dknzx"] Aug 04 15:57:01.865586 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.865495 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:01.865644 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:01.865612 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:57:01.869148 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.869122 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ltd4v"] Aug 04 15:57:01.869310 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.869230 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:01.869388 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:01.869338 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:57:01.869830 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.869810 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6xd6d"] Aug 04 15:57:01.869933 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.869892 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:01.869992 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:01.869959 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:57:01.942892 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:01.942809 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:02.043722 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:02.043688 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:02.390365 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:02.390327 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:02.491354 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:02.491312 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:03.357178 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:03.357150 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:03.357353 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:03.357151 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:03.357353 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:03.357154 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:03.357353 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:03.357284 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:57:03.357503 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:03.357371 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:57:03.357503 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:03.357449 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:57:03.425669 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:03.425634 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:03.525890 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:03.525861 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:03.626360 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:03.626334 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:04.376421 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:04.376380 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:04.456110 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:04.456076 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:04.481890 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:04.481863 2571 generic.go:356] "Generic (PLEG): container finished" podID="a32c5e48-78e8-4816-80c3-2d38d50a359f" containerID="42496b441f9f4433aec9489a96400e0418488a74a72f23c622b1383c6c5247ee" exitCode=0 Aug 04 15:57:04.482010 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:04.481902 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerDied","Data":"42496b441f9f4433aec9489a96400e0418488a74a72f23c622b1383c6c5247ee"} Aug 04 15:57:05.029673 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.029633 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:05.029849 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.029780 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:57:05.029849 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.029848 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret podName:818a2360-3c98-4f8d-b540-d1c90b7d1113 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:37.029832202 +0000 UTC m=+66.123432429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret") pod "global-pull-secret-syncer-dknzx" (UID: "818a2360-3c98-4f8d-b540-d1c90b7d1113") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:57:05.130347 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.130314 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:05.130478 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.130458 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:57:05.130530 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.130515 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:57:37.130501269 +0000 UTC m=+66.224101497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:57:05.231723 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.231692 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:05.231886 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.231833 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:57:05.231886 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.231855 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:57:05.231886 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.231866 2571 projected.go:196] Error preparing data for projected volume kube-api-access-d8m4k for pod openshift-network-diagnostics/network-check-target-6xd6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:57:05.231998 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.231930 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k podName:77e34eb5-f2d6-4f4b-bb61-78fed95fa688 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:37.231909628 +0000 UTC m=+66.325509860 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d8m4k" (UniqueName: "kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k") pod "network-check-target-6xd6d" (UID: "77e34eb5-f2d6-4f4b-bb61-78fed95fa688") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:57:05.357002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.356981 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:05.357098 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.356980 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:05.357137 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.357095 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:57:05.357245 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.357230 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:05.357313 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.357229 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:57:05.357359 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:05.357299 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:57:05.486760 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.486722 2571 generic.go:356] "Generic (PLEG): container finished" podID="a32c5e48-78e8-4816-80c3-2d38d50a359f" containerID="1bf35536fce84f7db2131874de381416adaa9216a5cd1a5a15bd885b858abeba" exitCode=0 Aug 04 15:57:05.487115 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:05.486776 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerDied","Data":"1bf35536fce84f7db2131874de381416adaa9216a5cd1a5a15bd885b858abeba"} Aug 04 15:57:06.364062 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:06.363870 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:07.357644 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.357601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:07.358099 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.357616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:07.358099 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:07.357803 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dknzx" podUID="818a2360-3c98-4f8d-b540-d1c90b7d1113" Aug 04 15:57:07.358099 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.357616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:07.358099 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:07.357834 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ltd4v" podUID="d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d" Aug 04 15:57:07.358099 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:07.357888 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xd6d" podUID="77e34eb5-f2d6-4f4b-bb61-78fed95fa688" Aug 04 15:57:07.707535 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.707493 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-14.ec2.internal" event="NodeReady" Aug 04 15:57:07.707726 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.707695 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 15:57:07.758429 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.758396 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mbxx9"] Aug 04 15:57:07.761784 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.761760 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-j94lw"] Aug 04 15:57:07.761901 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.761876 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.764995 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.764973 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:07.765106 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.765062 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 15:57:07.765171 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.765120 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fnzc4\"" Aug 04 15:57:07.768044 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.768022 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 15:57:07.768155 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.768089 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-752h4\"" Aug 04 15:57:07.769116 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.769029 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 15:57:07.771851 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.771832 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 15:57:07.771945 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.771856 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 15:57:07.787144 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.787088 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mbxx9"] Aug 04 15:57:07.787914 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.787858 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j94lw"] Aug 04 15:57:07.848833 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.848794 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqr9\" (UniqueName: \"kubernetes.io/projected/fc13c2fb-981d-4308-8680-a1d97241dbee-kube-api-access-prqr9\") pod \"ingress-canary-j94lw\" (UID: \"fc13c2fb-981d-4308-8680-a1d97241dbee\") " pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:07.849021 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.848866 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955984cd-2260-47fe-a83e-1986fa17419a-tmp-dir\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.849021 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.848892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bmd\" (UniqueName: \"kubernetes.io/projected/955984cd-2260-47fe-a83e-1986fa17419a-kube-api-access-p8bmd\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.849021 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.848943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.849021 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.848975 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc13c2fb-981d-4308-8680-a1d97241dbee-cert\") pod \"ingress-canary-j94lw\" (UID: \"fc13c2fb-981d-4308-8680-a1d97241dbee\") " pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:07.849257 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.849019 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/955984cd-2260-47fe-a83e-1986fa17419a-config-volume\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.949462 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.949424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqr9\" (UniqueName: \"kubernetes.io/projected/fc13c2fb-981d-4308-8680-a1d97241dbee-kube-api-access-prqr9\") pod \"ingress-canary-j94lw\" (UID: \"fc13c2fb-981d-4308-8680-a1d97241dbee\") " pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:07.949642 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.949492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955984cd-2260-47fe-a83e-1986fa17419a-tmp-dir\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.949642 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.949517 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bmd\" (UniqueName: \"kubernetes.io/projected/955984cd-2260-47fe-a83e-1986fa17419a-kube-api-access-p8bmd\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.949642 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.949558 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.949642 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.949582 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc13c2fb-981d-4308-8680-a1d97241dbee-cert\") pod \"ingress-canary-j94lw\" (UID: \"fc13c2fb-981d-4308-8680-a1d97241dbee\") " pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:07.949642 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.949606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/955984cd-2260-47fe-a83e-1986fa17419a-config-volume\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.949869 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:07.949721 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:07.949869 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:07.949805 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls podName:955984cd-2260-47fe-a83e-1986fa17419a nodeName:}" failed. No retries permitted until 2026-08-04 15:57:08.449783633 +0000 UTC m=+37.543383872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls") pod "dns-default-mbxx9" (UID: "955984cd-2260-47fe-a83e-1986fa17419a") : secret "dns-default-metrics-tls" not found Aug 04 15:57:07.950033 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.950010 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/955984cd-2260-47fe-a83e-1986fa17419a-tmp-dir\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.950325 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.950303 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/955984cd-2260-47fe-a83e-1986fa17419a-config-volume\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.954348 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.954324 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc13c2fb-981d-4308-8680-a1d97241dbee-cert\") pod \"ingress-canary-j94lw\" (UID: \"fc13c2fb-981d-4308-8680-a1d97241dbee\") " pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:07.959270 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.959168 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bmd\" (UniqueName: \"kubernetes.io/projected/955984cd-2260-47fe-a83e-1986fa17419a-kube-api-access-p8bmd\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:07.959270 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.959177 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqr9\" (UniqueName: \"kubernetes.io/projected/fc13c2fb-981d-4308-8680-a1d97241dbee-kube-api-access-prqr9\") pod \"ingress-canary-j94lw\" (UID: \"fc13c2fb-981d-4308-8680-a1d97241dbee\") " pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:07.973310 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:07.973281 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:08.049528 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:08.049486 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:08.079380 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:08.079354 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-j94lw" Aug 04 15:57:08.224291 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:08.224181 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-j94lw"] Aug 04 15:57:08.452653 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:08.452607 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:08.453565 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:08.452797 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:08.453565 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:08.452883 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls podName:955984cd-2260-47fe-a83e-1986fa17419a nodeName:}" failed. No retries permitted until 2026-08-04 15:57:09.452860907 +0000 UTC m=+38.546461137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls") pod "dns-default-mbxx9" (UID: "955984cd-2260-47fe-a83e-1986fa17419a") : secret "dns-default-metrics-tls" not found Aug 04 15:57:09.357518 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.357484 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:09.357677 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.357532 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:09.357677 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.357484 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:09.362110 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.362071 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 15:57:09.362271 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.362253 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bfprq\"" Aug 04 15:57:09.363383 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.363359 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 15:57:09.363479 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.363428 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xj8nj\"" Aug 04 15:57:09.363540 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.363433 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 15:57:09.363540 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.363530 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 15:57:09.461896 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:09.461854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:09.462310 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:09.461986 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:09.462310 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:09.462067 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls podName:955984cd-2260-47fe-a83e-1986fa17419a nodeName:}" failed. No retries permitted until 2026-08-04 15:57:11.462048621 +0000 UTC m=+40.555648848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls") pod "dns-default-mbxx9" (UID: "955984cd-2260-47fe-a83e-1986fa17419a") : secret "dns-default-metrics-tls" not found Aug 04 15:57:09.656063 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:09.655944 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc13c2fb_981d_4308_8680_a1d97241dbee.slice/crio-be2c4ef9c9681888625f470ff4609878b9a66c5a23764700095248a7e48a00f9 WatchSource:0}: Error finding container be2c4ef9c9681888625f470ff4609878b9a66c5a23764700095248a7e48a00f9: Status 404 returned error can't find the container with id be2c4ef9c9681888625f470ff4609878b9a66c5a23764700095248a7e48a00f9 Aug 04 15:57:10.501942 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:10.501908 2571 generic.go:356] "Generic (PLEG): container finished" podID="a32c5e48-78e8-4816-80c3-2d38d50a359f" containerID="f8e4b4286204d59038460604978f6ea824963446a45cc0285d05bb636c3c5a7a" exitCode=0 Aug 04 15:57:10.502371 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:10.501984 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerDied","Data":"f8e4b4286204d59038460604978f6ea824963446a45cc0285d05bb636c3c5a7a"} Aug 04 15:57:10.503077 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:10.503058 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j94lw" event={"ID":"fc13c2fb-981d-4308-8680-a1d97241dbee","Type":"ContainerStarted","Data":"be2c4ef9c9681888625f470ff4609878b9a66c5a23764700095248a7e48a00f9"} Aug 04 15:57:11.477155 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:11.477123 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:11.477351 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:11.477270 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:11.477351 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:11.477331 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls podName:955984cd-2260-47fe-a83e-1986fa17419a nodeName:}" failed. No retries permitted until 2026-08-04 15:57:15.477316472 +0000 UTC m=+44.570916698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls") pod "dns-default-mbxx9" (UID: "955984cd-2260-47fe-a83e-1986fa17419a") : secret "dns-default-metrics-tls" not found Aug 04 15:57:11.509152 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:11.509113 2571 generic.go:356] "Generic (PLEG): container finished" podID="a32c5e48-78e8-4816-80c3-2d38d50a359f" containerID="d0c188544d5bf704dadf340f03cce8c76aedab843d882617c73b06e5a1e78efb" exitCode=0 Aug 04 15:57:11.509616 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:11.509177 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerDied","Data":"d0c188544d5bf704dadf340f03cce8c76aedab843d882617c73b06e5a1e78efb"} Aug 04 15:57:11.701527 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:11.701430 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:11.794755 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:11.794642 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:12.515083 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:12.515009 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" event={"ID":"a32c5e48-78e8-4816-80c3-2d38d50a359f","Type":"ContainerStarted","Data":"07fe678439d6ec5f4efbcf594e4659d9266f2372deaf4a152bf124191e4d7c42"} Aug 04 15:57:12.548624 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:12.548574 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6zhbq" podStartSLOduration=7.051619503 podStartE2EDuration="40.548556653s" totalImagesPullingTime="33.49693715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:12.547897315 +0000 UTC m=+41.641497563" watchObservedRunningTime="2026-08-04 15:57:12.548556653 +0000 UTC m=+41.642156900" Aug 04 15:57:12.721849 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:12.721813 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:12.818976 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:12.818944 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:12.890178 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:12.890128 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:13.519351 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:13.519317 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-j94lw" event={"ID":"fc13c2fb-981d-4308-8680-a1d97241dbee","Type":"ContainerStarted","Data":"6ce032375b3cdbd413e2612582fb24720eb728744df5c7fd3e231d0d22aeef1b"} Aug 04 15:57:13.540731 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:13.540679 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-j94lw" podStartSLOduration=3.599943341 podStartE2EDuration="6.540663042s" totalImagesPullingTime="2.940719701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:13.539906161 +0000 UTC m=+42.633506411" watchObservedRunningTime="2026-08-04 15:57:13.540663042 +0000 UTC m=+42.634263499" Aug 04 15:57:13.645395 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:13.645343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:13.745545 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:13.745515 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:13.824396 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:13.824366 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:15.055860 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:15.055819 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:15.154770 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:15.154731 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:15.507332 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:15.507250 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:15.507472 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:15.507372 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:15.507472 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:15.507425 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls podName:955984cd-2260-47fe-a83e-1986fa17419a nodeName:}" failed. No retries permitted until 2026-08-04 15:57:23.507411261 +0000 UTC m=+52.601011487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls") pod "dns-default-mbxx9" (UID: "955984cd-2260-47fe-a83e-1986fa17419a") : secret "dns-default-metrics-tls" not found Aug 04 15:57:20.367833 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:20.367802 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:20.471214 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:20.471163 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:21.568820 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:21.568788 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:21.697843 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:21.697815 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:23.562420 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:23.562385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:23.562882 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:23.562519 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:57:23.562882 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:23.562587 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls podName:955984cd-2260-47fe-a83e-1986fa17419a nodeName:}" failed. No retries permitted until 2026-08-04 15:57:39.562567586 +0000 UTC m=+68.656167815 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls") pod "dns-default-mbxx9" (UID: "955984cd-2260-47fe-a83e-1986fa17419a") : secret "dns-default-metrics-tls" not found Aug 04 15:57:24.767526 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.767492 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5"] Aug 04 15:57:24.788153 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.788124 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5"] Aug 04 15:57:24.788308 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.788245 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.794358 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.794336 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 15:57:24.794487 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.794339 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 15:57:24.794487 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.794462 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 15:57:24.794669 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.794647 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 15:57:24.794731 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.794707 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-956c7\"" Aug 04 15:57:24.870915 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.870885 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/79dae8ed-e996-4bcb-bdfb-862526c96a37-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.870915 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.870920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwp57\" (UniqueName: \"kubernetes.io/projected/79dae8ed-e996-4bcb-bdfb-862526c96a37-kube-api-access-zwp57\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.871112 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.871008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.876361 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.876330 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd"] Aug 04 15:57:24.900010 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.899984 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv"] Aug 04 15:57:24.900155 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.900106 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:24.903153 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.903135 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 15:57:24.903299 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.903136 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 15:57:24.903459 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.903432 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jf7f7\"" Aug 04 15:57:24.903673 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.903656 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 15:57:24.903718 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.903669 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:57:24.920034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.920017 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd"] Aug 04 15:57:24.920131 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.920046 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv"] Aug 04 15:57:24.920131 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.920111 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:24.923187 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.923167 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 15:57:24.925033 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.924055 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 15:57:24.925033 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.924601 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:57:24.925033 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.924802 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-x8p6q\"" Aug 04 15:57:24.925033 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.924823 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 15:57:24.971561 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.971531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/79dae8ed-e996-4bcb-bdfb-862526c96a37-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.971561 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.971563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwp57\" (UniqueName: \"kubernetes.io/projected/79dae8ed-e996-4bcb-bdfb-862526c96a37-kube-api-access-zwp57\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.971793 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.971605 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f542629-2dd8-4570-ad36-652186645e9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:24.971793 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.971632 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.971793 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.971651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f542629-2dd8-4570-ad36-652186645e9e-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:24.971793 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.971666 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlmk\" (UniqueName: \"kubernetes.io/projected/6f542629-2dd8-4570-ad36-652186645e9e-kube-api-access-5vlmk\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:24.971793 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:24.971764 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:24.972049 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:24.971829 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls podName:79dae8ed-e996-4bcb-bdfb-862526c96a37 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:25.471815553 +0000 UTC m=+54.565415778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-jmlx5" (UID: "79dae8ed-e996-4bcb-bdfb-862526c96a37") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:24.972111 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.972070 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-blmkg"] Aug 04 15:57:24.972422 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.972401 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/79dae8ed-e996-4bcb-bdfb-862526c96a37-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.993661 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.993638 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwp57\" (UniqueName: \"kubernetes.io/projected/79dae8ed-e996-4bcb-bdfb-862526c96a37-kube-api-access-zwp57\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:24.994118 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.994097 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7994b49464-qzkr9"] Aug 04 15:57:24.994237 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.994222 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:24.996979 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.996955 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 15:57:24.997481 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.997459 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 15:57:24.997785 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.997771 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 15:57:24.998500 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:24.998477 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 15:57:25.001411 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.001389 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qsj9t\"" Aug 04 15:57:25.005841 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.005818 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 15:57:25.012345 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.012329 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-blmkg"] Aug 04 15:57:25.012429 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.012356 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7994b49464-qzkr9"] Aug 04 15:57:25.012466 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.012434 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.015469 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.015455 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l556n\"" Aug 04 15:57:25.015652 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.015636 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 15:57:25.015711 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.015654 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 15:57:25.015711 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.015688 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 15:57:25.022826 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.022754 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 15:57:25.072079 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072044 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f542629-2dd8-4570-ad36-652186645e9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:25.072079 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072077 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhg6\" (UniqueName: \"kubernetes.io/projected/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-kube-api-access-rnhg6\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.072325 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072100 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f542629-2dd8-4570-ad36-652186645e9e-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:25.072325 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-certificates\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.072325 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072282 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-image-registry-private-configuration\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.072325 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072316 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/023f756f-dd53-42b9-a4ab-7411cb56ecef-ca-trust-extracted\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.072453 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072397 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-serving-cert\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.072453 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072426 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c856p\" (UniqueName: \"kubernetes.io/projected/5955abed-5484-4f74-a6ff-8490cc3114b3-kube-api-access-c856p\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.072453 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072446 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-config\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.072570 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072462 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5955abed-5484-4f74-a6ff-8490cc3114b3-trusted-ca-bundle\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.072570 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072509 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5955abed-5484-4f74-a6ff-8490cc3114b3-snapshots\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.072570 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlmk\" (UniqueName: \"kubernetes.io/projected/6f542629-2dd8-4570-ad36-652186645e9e-kube-api-access-5vlmk\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:25.072695 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072571 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-trusted-ca\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.072736 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-bound-sa-token\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.072736 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072705 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f542629-2dd8-4570-ad36-652186645e9e-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:25.072736 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072731 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5955abed-5484-4f74-a6ff-8490cc3114b3-serving-cert\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.072846 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2tp\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-kube-api-access-9n2tp\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.072846 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072829 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5955abed-5484-4f74-a6ff-8490cc3114b3-tmp\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.072922 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072845 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-installation-pull-secrets\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.072922 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072892 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5955abed-5484-4f74-a6ff-8490cc3114b3-service-ca-bundle\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.072922 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.072915 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.074425 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.074404 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f542629-2dd8-4570-ad36-652186645e9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:25.084877 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.084855 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlmk\" (UniqueName: \"kubernetes.io/projected/6f542629-2dd8-4570-ad36-652186645e9e-kube-api-access-5vlmk\") pod \"kube-storage-version-migrator-operator-7878f4b88c-w2qqd\" (UID: \"6f542629-2dd8-4570-ad36-652186645e9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:25.125898 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.125863 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:25.173723 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173691 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-certificates\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.173723 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173727 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-image-registry-private-configuration\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.173942 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173747 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/023f756f-dd53-42b9-a4ab-7411cb56ecef-ca-trust-extracted\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.173942 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-serving-cert\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.173942 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c856p\" (UniqueName: \"kubernetes.io/projected/5955abed-5484-4f74-a6ff-8490cc3114b3-kube-api-access-c856p\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.173942 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173879 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-config\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.173942 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5955abed-5484-4f74-a6ff-8490cc3114b3-trusted-ca-bundle\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173947 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5955abed-5484-4f74-a6ff-8490cc3114b3-snapshots\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.173985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-trusted-ca\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174025 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-bound-sa-token\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174047 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5955abed-5484-4f74-a6ff-8490cc3114b3-serving-cert\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174079 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2tp\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-kube-api-access-9n2tp\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174106 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5955abed-5484-4f74-a6ff-8490cc3114b3-tmp\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-installation-pull-secrets\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174162 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5955abed-5484-4f74-a6ff-8490cc3114b3-service-ca-bundle\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174176 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/023f756f-dd53-42b9-a4ab-7411cb56ecef-ca-trust-extracted\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.174195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.174712 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhg6\" (UniqueName: \"kubernetes.io/projected/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-kube-api-access-rnhg6\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.174712 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174504 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-config\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.174950 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174926 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5955abed-5484-4f74-a6ff-8490cc3114b3-trusted-ca-bundle\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.175023 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.174968 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-certificates\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.175095 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.175073 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5955abed-5484-4f74-a6ff-8490cc3114b3-tmp\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.175154 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.175091 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-trusted-ca\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.175311 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.175292 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:57:25.175386 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.175313 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7994b49464-qzkr9: secret "image-registry-tls" not found Aug 04 15:57:25.175386 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.175376 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls podName:023f756f-dd53-42b9-a4ab-7411cb56ecef nodeName:}" failed. No retries permitted until 2026-08-04 15:57:25.675357667 +0000 UTC m=+54.768957911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls") pod "image-registry-7994b49464-qzkr9" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef") : secret "image-registry-tls" not found Aug 04 15:57:25.175518 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.175495 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/5955abed-5484-4f74-a6ff-8490cc3114b3-snapshots\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.175606 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.175585 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5955abed-5484-4f74-a6ff-8490cc3114b3-service-ca-bundle\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.176534 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.176508 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-serving-cert\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.176534 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.176530 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-image-registry-private-configuration\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.176662 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.176561 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5955abed-5484-4f74-a6ff-8490cc3114b3-serving-cert\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.177134 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.177107 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-installation-pull-secrets\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.183087 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.183061 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c856p\" (UniqueName: \"kubernetes.io/projected/5955abed-5484-4f74-a6ff-8490cc3114b3-kube-api-access-c856p\") pod \"insights-operator-77696685b-blmkg\" (UID: \"5955abed-5484-4f74-a6ff-8490cc3114b3\") " pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.183707 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.183689 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhg6\" (UniqueName: \"kubernetes.io/projected/a978fb25-a6d5-464b-a72f-bef0c4a60b4e-kube-api-access-rnhg6\") pod \"service-ca-operator-784f97bcf6-cgksv\" (UID: \"a978fb25-a6d5-464b-a72f-bef0c4a60b4e\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.183800 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.183751 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-bound-sa-token\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.184127 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.184111 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2tp\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-kube-api-access-9n2tp\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.208951 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.208927 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" Aug 04 15:57:25.227002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.226975 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:25.231839 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.231822 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" Aug 04 15:57:25.306869 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.306779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-blmkg" Aug 04 15:57:25.363059 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.363039 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd"] Aug 04 15:57:25.365952 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:25.365920 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f542629_2dd8_4570_ad36_652186645e9e.slice/crio-59dd4fff714a7b9bb708b97c797b35bfb9c2f39d1022a588344db235a529df48 WatchSource:0}: Error finding container 59dd4fff714a7b9bb708b97c797b35bfb9c2f39d1022a588344db235a529df48: Status 404 returned error can't find the container with id 59dd4fff714a7b9bb708b97c797b35bfb9c2f39d1022a588344db235a529df48 Aug 04 15:57:25.407858 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.405303 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv"] Aug 04 15:57:25.477831 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.477791 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:25.478010 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.477963 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:25.478083 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.478046 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls podName:79dae8ed-e996-4bcb-bdfb-862526c96a37 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:26.478023572 +0000 UTC m=+55.571623813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-jmlx5" (UID: "79dae8ed-e996-4bcb-bdfb-862526c96a37") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:25.481629 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.481601 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-blmkg"] Aug 04 15:57:25.484602 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:25.484581 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5955abed_5484_4f74_a6ff_8490cc3114b3.slice/crio-4bbaa897da57aa4cbc687cfb3c129452469b1404522846616a31d162aca091f9 WatchSource:0}: Error finding container 4bbaa897da57aa4cbc687cfb3c129452469b1404522846616a31d162aca091f9: Status 404 returned error can't find the container with id 4bbaa897da57aa4cbc687cfb3c129452469b1404522846616a31d162aca091f9 Aug 04 15:57:25.548265 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.548133 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" event={"ID":"6f542629-2dd8-4570-ad36-652186645e9e","Type":"ContainerStarted","Data":"59dd4fff714a7b9bb708b97c797b35bfb9c2f39d1022a588344db235a529df48"} Aug 04 15:57:25.549366 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.549333 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-blmkg" event={"ID":"5955abed-5484-4f74-a6ff-8490cc3114b3","Type":"ContainerStarted","Data":"4bbaa897da57aa4cbc687cfb3c129452469b1404522846616a31d162aca091f9"} Aug 04 15:57:25.550387 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.550365 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" event={"ID":"a978fb25-a6d5-464b-a72f-bef0c4a60b4e","Type":"ContainerStarted","Data":"5481a9095b0d57dd60dc4b154a977a921cbda666994a47476226152d1d0d8b08"} Aug 04 15:57:25.679568 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:25.679525 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:25.679770 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.679678 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:57:25.679770 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.679701 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7994b49464-qzkr9: secret "image-registry-tls" not found Aug 04 15:57:25.679874 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:25.679818 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls podName:023f756f-dd53-42b9-a4ab-7411cb56ecef nodeName:}" failed. No retries permitted until 2026-08-04 15:57:26.6797974 +0000 UTC m=+55.773397626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls") pod "image-registry-7994b49464-qzkr9" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef") : secret "image-registry-tls" not found Aug 04 15:57:26.485773 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:26.485726 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:26.486172 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:26.485869 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:26.486172 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:26.485935 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls podName:79dae8ed-e996-4bcb-bdfb-862526c96a37 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:28.485918708 +0000 UTC m=+57.579518937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-jmlx5" (UID: "79dae8ed-e996-4bcb-bdfb-862526c96a37") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:26.687232 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:26.687175 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:26.687419 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:26.687322 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:57:26.687419 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:26.687348 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7994b49464-qzkr9: secret "image-registry-tls" not found Aug 04 15:57:26.687419 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:26.687413 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls podName:023f756f-dd53-42b9-a4ab-7411cb56ecef nodeName:}" failed. No retries permitted until 2026-08-04 15:57:28.687394023 +0000 UTC m=+57.780994251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls") pod "image-registry-7994b49464-qzkr9" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef") : secret "image-registry-tls" not found Aug 04 15:57:27.556109 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:27.556069 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" event={"ID":"a978fb25-a6d5-464b-a72f-bef0c4a60b4e","Type":"ContainerStarted","Data":"c60308bbc74ddc0fc4d264e60f32495321e5778b5ae97403b52280fb7ae84105"} Aug 04 15:57:27.585146 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:27.585093 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" podStartSLOduration=1.829487911 podStartE2EDuration="3.585075373s" totalImagesPullingTime="1.755587462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:27.584508297 +0000 UTC m=+56.678108546" watchObservedRunningTime="2026-08-04 15:57:27.585075373 +0000 UTC m=+56.678675615" Aug 04 15:57:28.501034 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:28.500990 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:28.501260 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:28.501155 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:28.501260 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:28.501235 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls podName:79dae8ed-e996-4bcb-bdfb-862526c96a37 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:32.501216182 +0000 UTC m=+61.594816422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-jmlx5" (UID: "79dae8ed-e996-4bcb-bdfb-862526c96a37") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:28.581646 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:28.581611 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:28.702164 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:28.702129 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:28.702363 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:28.702337 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:57:28.702410 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:28.702369 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7994b49464-qzkr9: secret "image-registry-tls" not found Aug 04 15:57:28.702444 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:28.702437 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls podName:023f756f-dd53-42b9-a4ab-7411cb56ecef nodeName:}" failed. No retries permitted until 2026-08-04 15:57:32.702417048 +0000 UTC m=+61.796017277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls") pod "image-registry-7994b49464-qzkr9" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef") : secret "image-registry-tls" not found Aug 04 15:57:28.972471 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:28.972432 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:29.055984 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:29.055947 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:29.443918 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:29.443852 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:29.558427 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:29.558370 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:29.677399 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:29.677358 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:29.780242 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:29.780132 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:30.565463 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:30.565426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" event={"ID":"6f542629-2dd8-4570-ad36-652186645e9e","Type":"ContainerStarted","Data":"ec4437e92853229bfcb137e214558224c2326be84e22cce615b41eb427fda582"} Aug 04 15:57:30.566840 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:30.566807 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-blmkg" event={"ID":"5955abed-5484-4f74-a6ff-8490cc3114b3","Type":"ContainerStarted","Data":"27fc3fe35f8d926f9b2a7779568d1ec60219a81622797790852b43a5daa2ef58"} Aug 04 15:57:30.583108 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:30.582955 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" podStartSLOduration=2.085131098 podStartE2EDuration="6.582937005s" totalImagesPullingTime="4.497805907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:30.582657949 +0000 UTC m=+59.676258198" watchObservedRunningTime="2026-08-04 15:57:30.582937005 +0000 UTC m=+59.676537255" Aug 04 15:57:30.603884 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:30.603829 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-blmkg" podStartSLOduration=2.224077531 podStartE2EDuration="6.603809038s" totalImagesPullingTime="4.379731507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:30.602395422 +0000 UTC m=+59.695995672" watchObservedRunningTime="2026-08-04 15:57:30.603809038 +0000 UTC m=+59.697409287" Aug 04 15:57:31.174488 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:31.174376 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:31.269961 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:31.269925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:31.365513 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:31.365471 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:32.043043 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:32.043008 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:32.542234 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:32.542178 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:32.542730 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:32.542354 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:32.542730 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:32.542464 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls podName:79dae8ed-e996-4bcb-bdfb-862526c96a37 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:40.542442404 +0000 UTC m=+69.636042634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-jmlx5" (UID: "79dae8ed-e996-4bcb-bdfb-862526c96a37") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:32.743906 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:32.743866 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:32.744063 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:32.744018 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:57:32.744063 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:32.744037 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7994b49464-qzkr9: secret "image-registry-tls" not found Aug 04 15:57:32.744138 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:32.744092 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls podName:023f756f-dd53-42b9-a4ab-7411cb56ecef nodeName:}" failed. No retries permitted until 2026-08-04 15:57:40.744076522 +0000 UTC m=+69.837676748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls") pod "image-registry-7994b49464-qzkr9" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef") : secret "image-registry-tls" not found Aug 04 15:57:33.487970 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:33.487941 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dsn4r" Aug 04 15:57:34.442360 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:34.442321 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:34.540876 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:34.540839 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:34.717002 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:34.716911 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:34.752006 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:34.751982 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vbw9c_ba50246f-15ba-4589-9ae4-f38e8169ceec/dns-node-resolver/0.log" Aug 04 15:57:34.843827 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:34.843793 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:34.989996 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:34.989913 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:35.217570 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:35.217538 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:35.750504 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:35.750477 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pxh8_e97926c6-57de-4b89-9fdb-fe0d7310fb68/node-ca/0.log" Aug 04 15:57:36.211979 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:36.211950 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:36.350113 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:36.350080 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j94lw_fc13c2fb-981d-4308-8680-a1d97241dbee/serve-healthcheck-canary/0.log" Aug 04 15:57:36.607438 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:36.607405 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:36.809468 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:36.809433 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:37.020125 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.020051 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:37.078291 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.078254 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:37.081530 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.081511 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 15:57:37.090643 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.090615 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/818a2360-3c98-4f8d-b540-d1c90b7d1113-original-pull-secret\") pod \"global-pull-secret-syncer-dknzx\" (UID: \"818a2360-3c98-4f8d-b540-d1c90b7d1113\") " pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:37.178793 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.178752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:57:37.181601 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.181576 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 15:57:37.189003 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:37.188974 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Aug 04 15:57:37.189134 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:37.189075 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs podName:d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d nodeName:}" failed. No retries permitted until 2026-08-04 15:58:41.189052281 +0000 UTC m=+130.282652507 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs") pod "network-metrics-daemon-ltd4v" (UID: "d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d") : secret "metrics-daemon-secret" not found Aug 04 15:57:37.210195 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.210166 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:37.277507 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.277424 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dknzx" Aug 04 15:57:37.279815 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.279793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:37.282683 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.282666 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 15:57:37.293338 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.293315 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 15:57:37.303274 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.303248 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8m4k\" (UniqueName: \"kubernetes.io/projected/77e34eb5-f2d6-4f4b-bb61-78fed95fa688-kube-api-access-d8m4k\") pod \"network-check-target-6xd6d\" (UID: \"77e34eb5-f2d6-4f4b-bb61-78fed95fa688\") " pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:37.356535 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.356508 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-w2qqd_6f542629-2dd8-4570-ad36-652186645e9e/kube-storage-version-migrator-operator/0.log" Aug 04 15:57:37.404458 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.404420 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dknzx"] Aug 04 15:57:37.408389 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:37.408358 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818a2360_3c98_4f8d_b540_d1c90b7d1113.slice/crio-ac41c8a4f96bb914882a15be9d1eb5ca834b080ba5189b7273250db1b4f56448 WatchSource:0}: Error finding container ac41c8a4f96bb914882a15be9d1eb5ca834b080ba5189b7273250db1b4f56448: Status 404 returned error can't find the container with id ac41c8a4f96bb914882a15be9d1eb5ca834b080ba5189b7273250db1b4f56448 Aug 04 15:57:37.585691 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.585652 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dknzx" event={"ID":"818a2360-3c98-4f8d-b540-d1c90b7d1113","Type":"ContainerStarted","Data":"ac41c8a4f96bb914882a15be9d1eb5ca834b080ba5189b7273250db1b4f56448"} Aug 04 15:57:37.585969 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.585955 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xj8nj\"" Aug 04 15:57:37.593067 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.593043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:37.713812 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:37.713664 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6xd6d"] Aug 04 15:57:37.716076 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:37.716053 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e34eb5_f2d6_4f4b_bb61_78fed95fa688.slice/crio-43e4f474b81234d170854bd188282c50238929d29819f119c62fcfef0baf0b47 WatchSource:0}: Error finding container 43e4f474b81234d170854bd188282c50238929d29819f119c62fcfef0baf0b47: Status 404 returned error can't find the container with id 43e4f474b81234d170854bd188282c50238929d29819f119c62fcfef0baf0b47 Aug 04 15:57:38.207952 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:38.207920 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:38.309460 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:38.309423 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:38.590794 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:38.590758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6xd6d" event={"ID":"77e34eb5-f2d6-4f4b-bb61-78fed95fa688","Type":"ContainerStarted","Data":"43e4f474b81234d170854bd188282c50238929d29819f119c62fcfef0baf0b47"} Aug 04 15:57:38.691135 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:38.691097 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:38.765546 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:38.765511 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:38.845931 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:38.845817 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:39.051452 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:39.051415 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:39.211486 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:39.211391 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:39.313344 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:39.313308 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:39.599590 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:39.599557 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:39.601901 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:39.601874 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/955984cd-2260-47fe-a83e-1986fa17419a-metrics-tls\") pod \"dns-default-mbxx9\" (UID: \"955984cd-2260-47fe-a83e-1986fa17419a\") " pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:39.877607 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:39.877526 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fnzc4\"" Aug 04 15:57:39.885294 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:39.885265 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:40.416845 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.416797 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:40.608569 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.608544 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:40.608723 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:40.608683 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:40.608786 ip-10-0-137-14 kubenswrapper[2571]: E0804 15:57:40.608747 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls podName:79dae8ed-e996-4bcb-bdfb-862526c96a37 nodeName:}" failed. No retries permitted until 2026-08-04 15:57:56.608727917 +0000 UTC m=+85.702328144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-jmlx5" (UID: "79dae8ed-e996-4bcb-bdfb-862526c96a37") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:57:40.609244 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.609172 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:40.624063 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.624042 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mbxx9"] Aug 04 15:57:40.626750 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:40.626728 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955984cd_2260_47fe_a83e_1986fa17419a.slice/crio-b23ed5aeebd262a94d6d2862ccebd2648bab4f6af4ac174ede96f7b9a5b90002 WatchSource:0}: Error finding container b23ed5aeebd262a94d6d2862ccebd2648bab4f6af4ac174ede96f7b9a5b90002: Status 404 returned error can't find the container with id b23ed5aeebd262a94d6d2862ccebd2648bab4f6af4ac174ede96f7b9a5b90002 Aug 04 15:57:40.810704 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.810667 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:40.813380 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.813348 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"image-registry-7994b49464-qzkr9\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:40.871282 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.871243 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:40.924018 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.923986 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l556n\"" Aug 04 15:57:40.931587 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.931559 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:40.980412 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:40.980374 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:41.083315 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.083193 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7994b49464-qzkr9"] Aug 04 15:57:41.087787 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:41.087757 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023f756f_dd53_42b9_a4ab_7411cb56ecef.slice/crio-1081396a0b0877c238d8a3be4de79d23e9201891cc384aeb0bbe01b55de98205 WatchSource:0}: Error finding container 1081396a0b0877c238d8a3be4de79d23e9201891cc384aeb0bbe01b55de98205: Status 404 returned error can't find the container with id 1081396a0b0877c238d8a3be4de79d23e9201891cc384aeb0bbe01b55de98205 Aug 04 15:57:41.125882 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.125808 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:41.222303 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.222187 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:41.602839 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.602788 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6xd6d" event={"ID":"77e34eb5-f2d6-4f4b-bb61-78fed95fa688","Type":"ContainerStarted","Data":"d502ad81d7222f50d04d2f1ae3abbd71e300006180cd62ebe70ebdc3824a0683"} Aug 04 15:57:41.603353 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.603061 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:57:41.607048 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.606664 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerStarted","Data":"4cc19df45a41dfecb2f264b210bbfa87bc35057beeb6f800d0b5e6d76e68226f"} Aug 04 15:57:41.607048 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.606714 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerStarted","Data":"1081396a0b0877c238d8a3be4de79d23e9201891cc384aeb0bbe01b55de98205"} Aug 04 15:57:41.607048 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.606770 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:57:41.608934 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.608870 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbxx9" event={"ID":"955984cd-2260-47fe-a83e-1986fa17419a","Type":"ContainerStarted","Data":"b23ed5aeebd262a94d6d2862ccebd2648bab4f6af4ac174ede96f7b9a5b90002"} Aug 04 15:57:41.622826 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.622648 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6xd6d" podStartSLOduration=66.785307916 podStartE2EDuration="1m9.622627631s" totalImagesPullingTime="2.837319715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:41.62116742 +0000 UTC m=+70.714767668" watchObservedRunningTime="2026-08-04 15:57:41.622627631 +0000 UTC m=+70.716227880" Aug 04 15:57:41.642522 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:41.642433 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podStartSLOduration=17.642417958 podStartE2EDuration="17.642417958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:41.641458046 +0000 UTC m=+70.735058294" watchObservedRunningTime="2026-08-04 15:57:41.642417958 +0000 UTC m=+70.736018184" Aug 04 15:57:42.614708 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:42.614664 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbxx9" event={"ID":"955984cd-2260-47fe-a83e-1986fa17419a","Type":"ContainerStarted","Data":"5c95094e1cb1fd0d5ea242e04da82f697eff7baae3c56061b114ee1418691150"} Aug 04 15:57:43.620162 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:43.620123 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbxx9" event={"ID":"955984cd-2260-47fe-a83e-1986fa17419a","Type":"ContainerStarted","Data":"2ba9cfaabd26bacdb662fa039c5e91fa2aeec926f502aaacfb44549768978e5c"} Aug 04 15:57:43.620670 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:43.620254 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:43.646437 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:43.646388 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mbxx9" podStartSLOduration=34.842048972 podStartE2EDuration="36.646369551s" totalImagesPullingTime="1.804320579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:43.646088816 +0000 UTC m=+72.739689061" watchObservedRunningTime="2026-08-04 15:57:43.646369551 +0000 UTC m=+72.739969799" Aug 04 15:57:44.090408 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:44.090326 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:44.190865 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:44.190825 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:44.262249 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:44.262136 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:44.603694 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:44.603657 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:44.802164 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:44.802129 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:45.009616 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:45.009537 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:45.206218 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:45.206171 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:45.348137 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:45.348101 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:45.628553 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:45.628466 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dknzx" event={"ID":"818a2360-3c98-4f8d-b540-d1c90b7d1113","Type":"ContainerStarted","Data":"df1b0bf0039b940e437b6ad3a71ece03a5248465a7a0eb69811b8a1ab95e8e70"} Aug 04 15:57:45.645439 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:45.645389 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dknzx" podStartSLOduration=65.991705614 podStartE2EDuration="1m13.645373007s" totalImagesPullingTime="7.653667393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:45.644836629 +0000 UTC m=+74.738436876" watchObservedRunningTime="2026-08-04 15:57:45.645373007 +0000 UTC m=+74.738973255" Aug 04 15:57:47.285795 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:47.285762 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:47.855324 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:47.855286 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:53.625650 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:53.625540 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mbxx9" Aug 04 15:57:56.633446 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:56.633401 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:56.635777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:56.635758 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79dae8ed-e996-4bcb-bdfb-862526c96a37-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-jmlx5\" (UID: \"79dae8ed-e996-4bcb-bdfb-862526c96a37\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:56.899356 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:56.899266 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-956c7\"" Aug 04 15:57:56.907081 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:56.907052 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" Aug 04 15:57:57.035436 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:57.035405 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5"] Aug 04 15:57:57.040357 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:57:57.038387 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79dae8ed_e996_4bcb_bdfb_862526c96a37.slice/crio-516aab0e531379a5f66fddfddbc8c6bc1108fb4815e93ed8a3abc5002c45fd07 WatchSource:0}: Error finding container 516aab0e531379a5f66fddfddbc8c6bc1108fb4815e93ed8a3abc5002c45fd07: Status 404 returned error can't find the container with id 516aab0e531379a5f66fddfddbc8c6bc1108fb4815e93ed8a3abc5002c45fd07 Aug 04 15:57:57.669390 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:57.669351 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" event={"ID":"79dae8ed-e996-4bcb-bdfb-862526c96a37","Type":"ContainerStarted","Data":"516aab0e531379a5f66fddfddbc8c6bc1108fb4815e93ed8a3abc5002c45fd07"} Aug 04 15:57:59.226658 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:59.226568 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:59.333278 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:59.333223 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:59.432505 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:59.432467 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:59.679301 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:59.679266 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" event={"ID":"79dae8ed-e996-4bcb-bdfb-862526c96a37","Type":"ContainerStarted","Data":"abbaccb0dd3c74acee71f0bbdd60e4573a9b1c4609106bb7b178b627dbdc6bc3"} Aug 04 15:57:59.704892 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:59.704837 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-jmlx5" podStartSLOduration=33.823105197 podStartE2EDuration="35.704822372s" totalImagesPullingTime="1.881717175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:57:59.704282915 +0000 UTC m=+88.797883160" watchObservedRunningTime="2026-08-04 15:57:59.704822372 +0000 UTC m=+88.798422645" Aug 04 15:57:59.752772 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:57:59.752733 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:00.936101 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:00.936064 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:00.936498 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:00.936123 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:02.619739 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:02.619704 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:02.620129 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:02.619763 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:07.991885 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:07.991846 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:08.099565 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:08.099528 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:08.226422 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:08.226393 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:08.407572 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:08.407534 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:08.477445 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:08.477397 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:08.825997 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:08.825965 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:08.905807 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:08.905771 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:09.036351 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:09.036316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:09.622768 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:09.622692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:09.829655 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:09.829618 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:09.982935 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:09.982828 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:10.456728 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:10.456695 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:10.531411 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:10.531383 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:10.936489 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:10.936453 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:10.936689 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:10.936548 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:12.617910 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:12.617875 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6xd6d" Aug 04 15:58:12.619549 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:12.619524 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:12.619637 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:12.619566 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:20.936510 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:20.936471 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:20.937060 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:20.936522 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:20.937060 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:20.936560 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:58:20.937060 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:20.936985 2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"4cc19df45a41dfecb2f264b210bbfa87bc35057beeb6f800d0b5e6d76e68226f"} pod="openshift-image-registry/image-registry-7994b49464-qzkr9" containerMessage="Container registry failed liveness probe, will be restarted" Aug 04 15:58:20.940442 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:20.940415 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:20.940568 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:20.940459 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:23.849347 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:23.849312 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:23.953223 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:23.953169 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:27.842570 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:27.842536 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:27.943481 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:27.943448 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:30.941271 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:30.941239 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:30.941730 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:30.941302 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:32.659182 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:32.659142 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:32.719029 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:32.718992 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:36.836777 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:36.836742 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:36.911510 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:36.911472 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:38.793137 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:38.793107 2571 generic.go:356] "Generic (PLEG): container finished" podID="a978fb25-a6d5-464b-a72f-bef0c4a60b4e" containerID="c60308bbc74ddc0fc4d264e60f32495321e5778b5ae97403b52280fb7ae84105" exitCode=0 Aug 04 15:58:38.793543 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:38.793151 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" event={"ID":"a978fb25-a6d5-464b-a72f-bef0c4a60b4e","Type":"ContainerDied","Data":"c60308bbc74ddc0fc4d264e60f32495321e5778b5ae97403b52280fb7ae84105"} Aug 04 15:58:38.793543 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:38.793495 2571 scope.go:117] "RemoveContainer" containerID="c60308bbc74ddc0fc4d264e60f32495321e5778b5ae97403b52280fb7ae84105" Aug 04 15:58:39.798575 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:39.798543 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-cgksv" event={"ID":"a978fb25-a6d5-464b-a72f-bef0c4a60b4e","Type":"ContainerStarted","Data":"d6c0c129ad9622ccc68e99b0954494a816c8032d22b0f0727a352d0f23b5e010"} Aug 04 15:58:40.802957 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:40.802929 2571 generic.go:356] "Generic (PLEG): container finished" podID="5955abed-5484-4f74-a6ff-8490cc3114b3" containerID="27fc3fe35f8d926f9b2a7779568d1ec60219a81622797790852b43a5daa2ef58" exitCode=0 Aug 04 15:58:40.803339 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:40.803010 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-blmkg" event={"ID":"5955abed-5484-4f74-a6ff-8490cc3114b3","Type":"ContainerDied","Data":"27fc3fe35f8d926f9b2a7779568d1ec60219a81622797790852b43a5daa2ef58"} Aug 04 15:58:40.803379 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:40.803351 2571 scope.go:117] "RemoveContainer" containerID="27fc3fe35f8d926f9b2a7779568d1ec60219a81622797790852b43a5daa2ef58" Aug 04 15:58:40.941694 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:40.941660 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:58:40.941864 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:40.941711 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:58:41.271166 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:41.271119 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:58:41.273456 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:41.273428 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d-metrics-certs\") pod \"network-metrics-daemon-ltd4v\" (UID: \"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d\") " pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:58:41.473175 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:41.473145 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bfprq\"" Aug 04 15:58:41.481892 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:41.481861 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ltd4v" Aug 04 15:58:41.623101 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:41.623070 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ltd4v"] Aug 04 15:58:41.626001 ip-10-0-137-14 kubenswrapper[2571]: W0804 15:58:41.625978 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6dd1c6f_ee03_4a25_b3cb_7187d9e2d78d.slice/crio-c993eb9a1aa192f7d4f03e5b3793652e375db3bb061de06c69e4f64f45103a4e WatchSource:0}: Error finding container c993eb9a1aa192f7d4f03e5b3793652e375db3bb061de06c69e4f64f45103a4e: Status 404 returned error can't find the container with id c993eb9a1aa192f7d4f03e5b3793652e375db3bb061de06c69e4f64f45103a4e Aug 04 15:58:41.808516 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:41.808485 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-blmkg" event={"ID":"5955abed-5484-4f74-a6ff-8490cc3114b3","Type":"ContainerStarted","Data":"9a7ac5944e603b6460a8ee484bebd080beda415d5dacc4da660b91fa735719dc"} Aug 04 15:58:41.809698 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:41.809674 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltd4v" event={"ID":"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d","Type":"ContainerStarted","Data":"c993eb9a1aa192f7d4f03e5b3793652e375db3bb061de06c69e4f64f45103a4e"} Aug 04 15:58:43.818421 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:43.818381 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltd4v" event={"ID":"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d","Type":"ContainerStarted","Data":"ff30f279068dde8304f55d614f15b294d8e66219fbecaa487ce86a805bf578d1"} Aug 04 15:58:43.818818 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:43.818448 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ltd4v" event={"ID":"d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d","Type":"ContainerStarted","Data":"be60f46eaced6a440416be274597da0711a65dee976d06df6c5b32de41426343"} Aug 04 15:58:43.836429 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:43.836376 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ltd4v" podStartSLOduration=130.594236166 podStartE2EDuration="2m11.83636155s" totalImagesPullingTime="1.242125384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:58:43.835273142 +0000 UTC m=+132.928873391" watchObservedRunningTime="2026-08-04 15:58:43.83636155 +0000 UTC m=+132.929961856" Aug 04 15:58:45.486681 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:45.486653 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbxx9_955984cd-2260-47fe-a83e-1986fa17419a/dns/0.log" Aug 04 15:58:45.686939 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:45.686907 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbxx9_955984cd-2260-47fe-a83e-1986fa17419a/kube-rbac-proxy/0.log" Aug 04 15:58:45.829138 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:45.829104 2571 generic.go:356] "Generic (PLEG): container finished" podID="6f542629-2dd8-4570-ad36-652186645e9e" containerID="ec4437e92853229bfcb137e214558224c2326be84e22cce615b41eb427fda582" exitCode=0 Aug 04 15:58:45.829315 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:45.829176 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" event={"ID":"6f542629-2dd8-4570-ad36-652186645e9e","Type":"ContainerDied","Data":"ec4437e92853229bfcb137e214558224c2326be84e22cce615b41eb427fda582"} Aug 04 15:58:45.829578 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:45.829559 2571 scope.go:117] "RemoveContainer" containerID="ec4437e92853229bfcb137e214558224c2326be84e22cce615b41eb427fda582" Aug 04 15:58:45.955217 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:45.955104 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" containerID="cri-o://4cc19df45a41dfecb2f264b210bbfa87bc35057beeb6f800d0b5e6d76e68226f" gracePeriod=30 Aug 04 15:58:46.086181 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.086155 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vbw9c_ba50246f-15ba-4589-9ae4-f38e8169ceec/dns-node-resolver/0.log" Aug 04 15:58:46.364924 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.364887 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:46.466303 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.466263 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:46.687158 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.687049 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7994b49464-qzkr9_023f756f-dd53-42b9-a4ab-7411cb56ecef/registry/0.log" Aug 04 15:58:46.833818 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.833776 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-w2qqd" event={"ID":"6f542629-2dd8-4570-ad36-652186645e9e","Type":"ContainerStarted","Data":"b583774b94ef0435899b79f405ed6e12506e68eb2a787d1a79100068fd43a05f"} Aug 04 15:58:46.835436 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.835409 2571 generic.go:356] "Generic (PLEG): container finished" podID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerID="4cc19df45a41dfecb2f264b210bbfa87bc35057beeb6f800d0b5e6d76e68226f" exitCode=0 Aug 04 15:58:46.835551 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.835441 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerDied","Data":"4cc19df45a41dfecb2f264b210bbfa87bc35057beeb6f800d0b5e6d76e68226f"} Aug 04 15:58:46.835551 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.835474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerStarted","Data":"df9347d63e752d9e26d159390b99a7b1811df1dc925208399346d8cd7bb71a3d"} Aug 04 15:58:46.835653 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:46.835639 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:58:47.087289 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:47.087263 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pxh8_e97926c6-57de-4b89-9fdb-fe0d7310fb68/node-ca/0.log" Aug 04 15:58:47.686158 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:47.686130 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j94lw_fc13c2fb-981d-4308-8680-a1d97241dbee/serve-healthcheck-canary/0.log" Aug 04 15:58:48.688974 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:48.688940 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-w2qqd_6f542629-2dd8-4570-ad36-652186645e9e/kube-storage-version-migrator-operator/1.log" Aug 04 15:58:55.911959 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:55.911918 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:55.977196 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:58:55.977160 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:00.937752 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:00.937714 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:00.938157 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:00.937790 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:05.456513 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:05.456479 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:05.711838 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:05.711698 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:07.843315 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:07.843277 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:07.843732 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:07.843330 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:10.935817 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:10.935774 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:10.936196 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:10.935842 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:15.056014 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:15.055976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:15.160324 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:15.160287 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:17.842663 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:17.842624 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:17.843120 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:17.842682 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:20.935613 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:20.935576 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:20.936047 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:20.935633 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:20.936047 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:20.935677 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:59:20.936167 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:20.936103 2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"df9347d63e752d9e26d159390b99a7b1811df1dc925208399346d8cd7bb71a3d"} pod="openshift-image-registry/image-registry-7994b49464-qzkr9" containerMessage="Container registry failed liveness probe, will be restarted" Aug 04 15:59:20.939816 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:20.939788 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:20.939959 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:20.939830 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:24.526534 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:24.526427 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:24.589692 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:24.589658 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:30.940701 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:30.940661 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:30.941170 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:30.940729 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:34.008899 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:34.008851 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:34.082056 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:34.082021 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:40.940808 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:40.940770 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:59:40.941184 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:40.940829 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:59:43.599000 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:43.598963 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:44.535959 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:44.535924 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:45.954234 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:45.954165 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" containerID="cri-o://df9347d63e752d9e26d159390b99a7b1811df1dc925208399346d8cd7bb71a3d" gracePeriod=30 Aug 04 15:59:47.024376 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:47.024340 2571 generic.go:356] "Generic (PLEG): container finished" podID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerID="df9347d63e752d9e26d159390b99a7b1811df1dc925208399346d8cd7bb71a3d" exitCode=0 Aug 04 15:59:47.024873 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:47.024407 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerDied","Data":"df9347d63e752d9e26d159390b99a7b1811df1dc925208399346d8cd7bb71a3d"} Aug 04 15:59:47.024873 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:47.024459 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerStarted","Data":"527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284"} Aug 04 15:59:47.024873 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:47.024483 2571 scope.go:117] "RemoveContainer" containerID="4cc19df45a41dfecb2f264b210bbfa87bc35057beeb6f800d0b5e6d76e68226f" Aug 04 15:59:47.024873 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:47.024692 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 15:59:53.123046 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:53.123012 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:53.229867 ip-10-0-137-14 kubenswrapper[2571]: I0804 15:59:53.229832 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:00.937271 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:00.937236 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:00:00.937675 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:00.937293 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:00:02.868451 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:02.868418 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:03.331615 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:03.331582 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:07.969079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:07.969032 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:08.034664 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:08.034631 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:00:08.034819 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:08.034681 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:00:08.060283 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:08.060250 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:10.022897 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:10.022864 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:10.936571 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:10.936527 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:00:10.936741 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:10.936582 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:00:12.126904 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:12.126869 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:12.187627 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:12.187594 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:18.033392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:18.033353 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:00:18.033758 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:18.033427 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:00:20.935739 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:20.935704 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:00:20.936176 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:20.935757 2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:00:20.936176 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:20.935797 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 16:00:20.936335 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:20.936241 2571 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284"} pod="openshift-image-registry/image-registry-7994b49464-qzkr9" containerMessage="Container registry failed liveness probe, will be restarted" Aug 04 16:00:20.939790 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:20.939766 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:00:20.939905 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:20.939809 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:00:21.649123 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:21.649075 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:21.755617 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:21.755579 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:30.940382 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:30.940344 2571 patch_prober.go:28] interesting pod/image-registry-7994b49464-qzkr9 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 16:00:30.940778 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:30.940420 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 16:00:31.336790 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.336759 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:31.613789 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.613707 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg"] Aug 04 16:00:31.616558 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.616531 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.619946 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.619595 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 04 16:00:31.622829 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.622803 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 16:00:31.622974 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.622847 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 16:00:31.624170 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.624144 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 16:00:31.631787 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.631764 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/58509348-4637-4dc8-afe6-2075f8b65e5f-klusterlet-config\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.631907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.631803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trnjs\" (UniqueName: \"kubernetes.io/projected/58509348-4637-4dc8-afe6-2075f8b65e5f-kube-api-access-trnjs\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.631907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.631835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58509348-4637-4dc8-afe6-2075f8b65e5f-tmp\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.657007 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.656979 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg"] Aug 04 16:00:31.716737 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.716702 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jdzkk"] Aug 04 16:00:31.719711 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.719688 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.721614 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.721480 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-854ddc5575-ng4bw"] Aug 04 16:00:31.725954 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.725927 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dngfj\"" Aug 04 16:00:31.728366 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.728346 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7994b49464-qzkr9"] Aug 04 16:00:31.728471 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.728451 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.732437 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732416 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 16:00:31.732541 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.732624 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732561 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9dqg\" (UniqueName: \"kubernetes.io/projected/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-kube-api-access-h9dqg\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.732624 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732582 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-data-volume\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.732717 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732639 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58509348-4637-4dc8-afe6-2075f8b65e5f-tmp\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.732717 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732667 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-crio-socket\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.732780 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/58509348-4637-4dc8-afe6-2075f8b65e5f-klusterlet-config\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.732816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732780 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.732816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732804 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trnjs\" (UniqueName: \"kubernetes.io/projected/58509348-4637-4dc8-afe6-2075f8b65e5f-kube-api-access-trnjs\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.732947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.732931 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58509348-4637-4dc8-afe6-2075f8b65e5f-tmp\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.735337 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.735222 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/58509348-4637-4dc8-afe6-2075f8b65e5f-klusterlet-config\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.736696 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.736673 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 16:00:31.737510 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.737492 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 16:00:31.737732 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.737711 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 16:00:31.737732 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.737726 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 16:00:31.743623 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.743596 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 16:00:31.743734 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.743664 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 16:00:31.743734 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.743701 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 16:00:31.743945 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.743927 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vmh6q\"" Aug 04 16:00:31.745055 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.745035 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdzkk"] Aug 04 16:00:31.745579 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.745562 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 16:00:31.750505 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.750488 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 16:00:31.762095 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.762071 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854ddc5575-ng4bw"] Aug 04 16:00:31.776001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.775969 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:31.785955 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.785935 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trnjs\" (UniqueName: \"kubernetes.io/projected/58509348-4637-4dc8-afe6-2075f8b65e5f-kube-api-access-trnjs\") pod \"klusterlet-addon-workmgr-646597956-8nbsg\" (UID: \"58509348-4637-4dc8-afe6-2075f8b65e5f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.833331 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833299 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-service-ca\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.833331 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833337 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-crio-socket\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833362 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-trusted-ca-bundle\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833413 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-oauth-serving-cert\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc6pw\" (UniqueName: \"kubernetes.io/projected/34309738-5c9b-44c1-a926-207b530e8440-kube-api-access-gc6pw\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-oauth-config\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833472 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-serving-cert\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833466 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-crio-socket\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.833560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.833946 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833616 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9dqg\" (UniqueName: \"kubernetes.io/projected/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-kube-api-access-h9dqg\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.833946 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833655 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-data-volume\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.833946 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.833704 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-console-config\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.834079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.834005 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-data-volume\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.834079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.834028 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.835894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.835870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.866318 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.866232 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44"] Aug 04 16:00:31.868581 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.868560 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9dqg\" (UniqueName: \"kubernetes.io/projected/3ce2d520-0b80-43f4-abac-1e74c4cb7af5-kube-api-access-h9dqg\") pod \"insights-runtime-extractor-jdzkk\" (UID: \"3ce2d520-0b80-43f4-abac-1e74c4cb7af5\") " pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:31.870479 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.870464 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" Aug 04 16:00:31.881387 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.881359 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-84jjm\"" Aug 04 16:00:31.881504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.881477 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Aug 04 16:00:31.891277 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.891249 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44"] Aug 04 16:00:31.909298 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.909272 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69558cbb7-9j4hh"] Aug 04 16:00:31.912162 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.912147 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.929467 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.929439 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:31.934096 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934071 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd632101-8637-4b0d-a51f-a3277af3b58e-ca-trust-extracted\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.934214 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd632101-8637-4b0d-a51f-a3277af3b58e-image-registry-private-configuration\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.934214 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-console-config\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.934214 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934175 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-service-ca\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.934398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934214 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd632101-8637-4b0d-a51f-a3277af3b58e-installation-pull-secrets\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.934398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934259 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-trusted-ca-bundle\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.934398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934282 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4r85\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-kube-api-access-r4r85\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.934398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-bound-sa-token\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.934398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934338 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/054e280d-3429-450e-87eb-0793502df9d5-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-kps44\" (UID: \"054e280d-3429-450e-87eb-0793502df9d5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" Aug 04 16:00:31.934398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-oauth-serving-cert\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.934398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc6pw\" (UniqueName: \"kubernetes.io/projected/34309738-5c9b-44c1-a926-207b530e8440-kube-api-access-gc6pw\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.934783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd632101-8637-4b0d-a51f-a3277af3b58e-registry-certificates\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.934783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-oauth-config\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.934783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934632 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-serving-cert\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.934783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934685 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-registry-tls\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.934783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd632101-8637-4b0d-a51f-a3277af3b58e-trusted-ca\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:31.935021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.934930 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-console-config\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.935173 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.935142 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-oauth-serving-cert\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.935173 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.935157 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-service-ca\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.935365 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.935348 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-trusted-ca-bundle\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.937114 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.937097 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-oauth-config\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.937404 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.937378 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-serving-cert\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:31.947540 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.947510 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:31.967346 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:31.967305 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc6pw\" (UniqueName: \"kubernetes.io/projected/34309738-5c9b-44c1-a926-207b530e8440-kube-api-access-gc6pw\") pod \"console-854ddc5575-ng4bw\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:32.030790 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.030762 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jdzkk" Aug 04 16:00:32.035071 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.034990 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd632101-8637-4b0d-a51f-a3277af3b58e-image-registry-private-configuration\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035071 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd632101-8637-4b0d-a51f-a3277af3b58e-installation-pull-secrets\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035071 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035043 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4r85\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-kube-api-access-r4r85\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035071 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035064 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-bound-sa-token\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035302 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035083 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/054e280d-3429-450e-87eb-0793502df9d5-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-kps44\" (UID: \"054e280d-3429-450e-87eb-0793502df9d5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" Aug 04 16:00:32.035302 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd632101-8637-4b0d-a51f-a3277af3b58e-registry-certificates\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035302 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-registry-tls\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035302 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035232 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd632101-8637-4b0d-a51f-a3277af3b58e-trusted-ca\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035302 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd632101-8637-4b0d-a51f-a3277af3b58e-ca-trust-extracted\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.035749 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.035687 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bd632101-8637-4b0d-a51f-a3277af3b58e-ca-trust-extracted\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.036710 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.036647 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bd632101-8637-4b0d-a51f-a3277af3b58e-trusted-ca\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.036920 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.036894 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bd632101-8637-4b0d-a51f-a3277af3b58e-registry-certificates\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.038417 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.038389 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bd632101-8637-4b0d-a51f-a3277af3b58e-image-registry-private-configuration\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.038417 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.038405 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/054e280d-3429-450e-87eb-0793502df9d5-tls-certificates\") pod \"prometheus-operator-admission-webhook-66685ff6d4-kps44\" (UID: \"054e280d-3429-450e-87eb-0793502df9d5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" Aug 04 16:00:32.038570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.038447 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bd632101-8637-4b0d-a51f-a3277af3b58e-installation-pull-secrets\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.038811 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.038791 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-registry-tls\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.043679 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.043653 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:32.057764 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.057739 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-bound-sa-token\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.063276 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.063249 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4r85\" (UniqueName: \"kubernetes.io/projected/bd632101-8637-4b0d-a51f-a3277af3b58e-kube-api-access-r4r85\") pod \"image-registry-69558cbb7-9j4hh\" (UID: \"bd632101-8637-4b0d-a51f-a3277af3b58e\") " pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.070081 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.070051 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69558cbb7-9j4hh"] Aug 04 16:00:32.093604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.093577 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg"] Aug 04 16:00:32.097037 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:32.097007 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58509348_4637_4dc8_afe6_2075f8b65e5f.slice/crio-0888c4950daaeade54cbcabaf5788b67167e56fc4a48b552e613a45795b5f9c0 WatchSource:0}: Error finding container 0888c4950daaeade54cbcabaf5788b67167e56fc4a48b552e613a45795b5f9c0: Status 404 returned error can't find the container with id 0888c4950daaeade54cbcabaf5788b67167e56fc4a48b552e613a45795b5f9c0 Aug 04 16:00:32.159933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.159898 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:32.171976 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.171946 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" event={"ID":"58509348-4637-4dc8-afe6-2075f8b65e5f","Type":"ContainerStarted","Data":"0888c4950daaeade54cbcabaf5788b67167e56fc4a48b552e613a45795b5f9c0"} Aug 04 16:00:32.182894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.182863 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" Aug 04 16:00:32.221101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.221063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:32.265717 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.265680 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jdzkk"] Aug 04 16:00:32.274264 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:32.274229 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce2d520_0b80_43f4_abac_1e74c4cb7af5.slice/crio-1cd7ab62968f5607cbd8840273e9e29220dedd8610c4068ca4a25ede98453a55 WatchSource:0}: Error finding container 1cd7ab62968f5607cbd8840273e9e29220dedd8610c4068ca4a25ede98453a55: Status 404 returned error can't find the container with id 1cd7ab62968f5607cbd8840273e9e29220dedd8610c4068ca4a25ede98453a55 Aug 04 16:00:32.319427 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.319403 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-854ddc5575-ng4bw"] Aug 04 16:00:32.342473 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:32.342443 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34309738_5c9b_44c1_a926_207b530e8440.slice/crio-63ed454f5f397dadaec99635d15bcce3b95b06991d0e2287661d4a8cc6420427 WatchSource:0}: Error finding container 63ed454f5f397dadaec99635d15bcce3b95b06991d0e2287661d4a8cc6420427: Status 404 returned error can't find the container with id 63ed454f5f397dadaec99635d15bcce3b95b06991d0e2287661d4a8cc6420427 Aug 04 16:00:32.365089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.365062 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44"] Aug 04 16:00:32.367162 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:32.367136 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054e280d_3429_450e_87eb_0793502df9d5.slice/crio-36f6e1d14a16083ee61013404db1748163883ef83e04f6cfbd3161e9c2358fb4 WatchSource:0}: Error finding container 36f6e1d14a16083ee61013404db1748163883ef83e04f6cfbd3161e9c2358fb4: Status 404 returned error can't find the container with id 36f6e1d14a16083ee61013404db1748163883ef83e04f6cfbd3161e9c2358fb4 Aug 04 16:00:32.408610 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.408579 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69558cbb7-9j4hh"] Aug 04 16:00:32.411923 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:32.411894 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd632101_8637_4b0d_a51f_a3277af3b58e.slice/crio-c6897db4ea28407b75da38cc636a4decf61a7e0139ab0a67a673803cb48606f6 WatchSource:0}: Error finding container c6897db4ea28407b75da38cc636a4decf61a7e0139ab0a67a673803cb48606f6: Status 404 returned error can't find the container with id c6897db4ea28407b75da38cc636a4decf61a7e0139ab0a67a673803cb48606f6 Aug 04 16:00:32.422617 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.422586 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:32.635051 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.634958 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:32.938092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:32.938011 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:33.183582 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.183535 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" event={"ID":"054e280d-3429-450e-87eb-0793502df9d5","Type":"ContainerStarted","Data":"36f6e1d14a16083ee61013404db1748163883ef83e04f6cfbd3161e9c2358fb4"} Aug 04 16:00:33.186768 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.186634 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdzkk" event={"ID":"3ce2d520-0b80-43f4-abac-1e74c4cb7af5","Type":"ContainerStarted","Data":"c5ed810d2c041e384a55648cad1a034f8ac3a06f1907db167d4ea6a38abd63f1"} Aug 04 16:00:33.186768 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.186702 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdzkk" event={"ID":"3ce2d520-0b80-43f4-abac-1e74c4cb7af5","Type":"ContainerStarted","Data":"5aa234da2011646fa29e1ab35f140c015c3ac2d68bcbf3dfac3e3ee478624eea"} Aug 04 16:00:33.186768 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.186732 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdzkk" event={"ID":"3ce2d520-0b80-43f4-abac-1e74c4cb7af5","Type":"ContainerStarted","Data":"1cd7ab62968f5607cbd8840273e9e29220dedd8610c4068ca4a25ede98453a55"} Aug 04 16:00:33.189536 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.189464 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" event={"ID":"bd632101-8637-4b0d-a51f-a3277af3b58e","Type":"ContainerStarted","Data":"57ffef00c7db4239bfaeb624158c5dd022f6d615819b9b39214f2396084c98a8"} Aug 04 16:00:33.189536 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.189518 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" event={"ID":"bd632101-8637-4b0d-a51f-a3277af3b58e","Type":"ContainerStarted","Data":"c6897db4ea28407b75da38cc636a4decf61a7e0139ab0a67a673803cb48606f6"} Aug 04 16:00:33.189930 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.189860 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:33.191588 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.191566 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854ddc5575-ng4bw" event={"ID":"34309738-5c9b-44c1-a926-207b530e8440","Type":"ContainerStarted","Data":"63ed454f5f397dadaec99635d15bcce3b95b06991d0e2287661d4a8cc6420427"} Aug 04 16:00:33.245449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:33.245382 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" podStartSLOduration=2.245362526 podStartE2EDuration="2.245362526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:33.245046856 +0000 UTC m=+242.338647095" watchObservedRunningTime="2026-08-04 16:00:33.245362526 +0000 UTC m=+242.338962776" Aug 04 16:00:34.201810 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:34.200186 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" event={"ID":"054e280d-3429-450e-87eb-0793502df9d5","Type":"ContainerStarted","Data":"34837a117b6253ea9ad91b73552c22e48f72e07109d4558b40b7721268d49fed"} Aug 04 16:00:34.201810 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:34.201505 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" Aug 04 16:00:34.207380 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:34.207333 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" Aug 04 16:00:34.261171 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:34.259963 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-66685ff6d4-kps44" podStartSLOduration=2.154283398 podStartE2EDuration="3.259942028s" totalImagesPullingTime="1.10565863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:34.223252792 +0000 UTC m=+243.316853040" watchObservedRunningTime="2026-08-04 16:00:34.259942028 +0000 UTC m=+243.353542278" Aug 04 16:00:34.552842 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:34.552654 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:34.667500 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:34.667409 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:34.909855 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:34.909702 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:37.713424 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:37.713383 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:38.217653 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.217610 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" event={"ID":"58509348-4637-4dc8-afe6-2075f8b65e5f","Type":"ContainerStarted","Data":"7a4297488014e0a0a9b98e51256d602d1d31d6feaaf8bc9817566e9b23045589"} Aug 04 16:00:38.217864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.217704 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:38.219883 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.219846 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jdzkk" event={"ID":"3ce2d520-0b80-43f4-abac-1e74c4cb7af5","Type":"ContainerStarted","Data":"90c09788bb4ec499a99fd790c15f8c53b5d91e81d71fdcb023b3b39a68791f91"} Aug 04 16:00:38.220002 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.219945 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" Aug 04 16:00:38.221452 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.221429 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854ddc5575-ng4bw" event={"ID":"34309738-5c9b-44c1-a926-207b530e8440","Type":"ContainerStarted","Data":"e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017"} Aug 04 16:00:38.237125 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.237083 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-646597956-8nbsg" podStartSLOduration=1.7470464479999999 podStartE2EDuration="7.237070517s" totalImagesPullingTime="5.490024069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:38.236587878 +0000 UTC m=+247.330188126" watchObservedRunningTime="2026-08-04 16:00:38.237070517 +0000 UTC m=+247.330670765" Aug 04 16:00:38.279462 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.279406 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jdzkk" podStartSLOduration=2.136118603 podStartE2EDuration="7.279387899s" totalImagesPullingTime="5.143269296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:00:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:38.278991347 +0000 UTC m=+247.372591595" watchObservedRunningTime="2026-08-04 16:00:38.279387899 +0000 UTC m=+247.372988147" Aug 04 16:00:38.301063 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.301008 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-854ddc5575-ng4bw" podStartSLOduration=2.062515735 podStartE2EDuration="7.300989721s" totalImagesPullingTime="5.238473986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:38.30073433 +0000 UTC m=+247.394334579" watchObservedRunningTime="2026-08-04 16:00:38.300989721 +0000 UTC m=+247.394589967" Aug 04 16:00:38.739947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.739902 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:38.978754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:38.978718 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.020606 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.020527 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s"] Aug 04 16:00:39.024125 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.024103 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.029762 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.029737 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Aug 04 16:00:39.029762 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.029754 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Aug 04 16:00:39.029940 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.029896 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ss5l5\"" Aug 04 16:00:39.038216 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.038185 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 16:00:39.044857 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.044834 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s"] Aug 04 16:00:39.049547 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.049526 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8hnkg"] Aug 04 16:00:39.052710 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.052690 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.055989 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.055970 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 16:00:39.056096 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.055975 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 16:00:39.056331 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.056307 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9rss6\"" Aug 04 16:00:39.056454 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.056438 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 16:00:39.097606 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.097606 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097607 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-sys\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.097782 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5cz5\" (UniqueName: \"kubernetes.io/projected/aaa34a74-128c-489d-9427-eda3783de4bd-kube-api-access-m5cz5\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.097782 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097655 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.097782 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097721 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b15423-4ffc-4055-9c27-68928094fff7-metrics-client-ca\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.097782 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.097936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gt46\" (UniqueName: \"kubernetes.io/projected/e6b15423-4ffc-4055-9c27-68928094fff7-kube-api-access-9gt46\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.097936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097840 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.097936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-wtmp\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.097936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097885 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-tls\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.097936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097908 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa34a74-128c-489d-9427-eda3783de4bd-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.097936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-root\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.098232 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.097947 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-textfile\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.125771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.125741 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.199041 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199002 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b15423-4ffc-4055-9c27-68928094fff7-metrics-client-ca\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199041 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199041 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.199328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199060 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gt46\" (UniqueName: \"kubernetes.io/projected/e6b15423-4ffc-4055-9c27-68928094fff7-kube-api-access-9gt46\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199328 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:39.199166 2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Aug 04 16:00:39.199328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199178 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199232 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-wtmp\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199328 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:39.199270 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls podName:aaa34a74-128c-489d-9427-eda3783de4bd nodeName:}" failed. No retries permitted until 2026-08-04 16:00:39.699249228 +0000 UTC m=+248.792849455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-lbn9s" (UID: "aaa34a74-128c-489d-9427-eda3783de4bd") : secret "openshift-state-metrics-tls" not found Aug 04 16:00:39.199328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-tls\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa34a74-128c-489d-9427-eda3783de4bd-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199367 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-wtmp\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199375 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-root\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199401 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-textfile\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199435 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-root\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199454 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-sys\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:39.199502 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:39.199620 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-tls podName:e6b15423-4ffc-4055-9c27-68928094fff7 nodeName:}" failed. No retries permitted until 2026-08-04 16:00:39.69960341 +0000 UTC m=+248.793203635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-tls") pod "node-exporter-8hnkg" (UID: "e6b15423-4ffc-4055-9c27-68928094fff7") : secret "node-exporter-tls" not found Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199554 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6b15423-4ffc-4055-9c27-68928094fff7-sys\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199520 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5cz5\" (UniqueName: \"kubernetes.io/projected/aaa34a74-128c-489d-9427-eda3783de4bd-kube-api-access-m5cz5\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.199727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b15423-4ffc-4055-9c27-68928094fff7-metrics-client-ca\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.200141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199743 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-textfile\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.200141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199789 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.200141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.199811 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.200141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.200016 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aaa34a74-128c-489d-9427-eda3783de4bd-metrics-client-ca\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.201726 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.201708 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.201824 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.201804 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.212871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.212837 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gt46\" (UniqueName: \"kubernetes.io/projected/e6b15423-4ffc-4055-9c27-68928094fff7-kube-api-access-9gt46\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.213257 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.213239 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5cz5\" (UniqueName: \"kubernetes.io/projected/aaa34a74-128c-489d-9427-eda3783de4bd-kube-api-access-m5cz5\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.233726 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.233692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.357507 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.357473 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.473506 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.473465 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.567554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.567522 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.704354 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.704268 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:39.704354 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.704326 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-tls\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.704579 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:39.704422 2571 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Aug 04 16:00:39.704579 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:39.704497 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls podName:aaa34a74-128c-489d-9427-eda3783de4bd nodeName:}" failed. No retries permitted until 2026-08-04 16:00:40.704482735 +0000 UTC m=+249.798082964 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls") pod "openshift-state-metrics-6bb4d7b4dd-lbn9s" (UID: "aaa34a74-128c-489d-9427-eda3783de4bd") : secret "openshift-state-metrics-tls" not found Aug 04 16:00:39.706681 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.706657 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6b15423-4ffc-4055-9c27-68928094fff7-node-exporter-tls\") pod \"node-exporter-8hnkg\" (UID: \"e6b15423-4ffc-4055-9c27-68928094fff7\") " pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.851001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.850962 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.961517 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.961423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hnkg" Aug 04 16:00:39.965676 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:39.965637 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:39.974967 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:39.974942 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b15423_4ffc_4055_9c27_68928094fff7.slice/crio-96efa4ec94b6803b329c53523d77aff6ec36952474c10e352e835692cb399cfe WatchSource:0}: Error finding container 96efa4ec94b6803b329c53523d77aff6ec36952474c10e352e835692cb399cfe: Status 404 returned error can't find the container with id 96efa4ec94b6803b329c53523d77aff6ec36952474c10e352e835692cb399cfe Aug 04 16:00:40.073251 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.073216 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:00:40.078117 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.078089 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:40.078283 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.078181 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.082542 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.082524 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 16:00:40.082662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.082561 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 16:00:40.082662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.082576 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 16:00:40.082662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.082570 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nnrzr\"" Aug 04 16:00:40.082662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.082566 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 16:00:40.082926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.082909 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 16:00:40.083022 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.083007 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 16:00:40.083088 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.083073 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 16:00:40.083140 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.083106 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 16:00:40.083230 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.083198 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 16:00:40.102598 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.102567 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:00:40.107468 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107442 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-volume\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107513 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107559 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107748 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107614 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107748 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107660 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-web-config\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107748 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107682 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107799 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107836 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhfc\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-kube-api-access-wlhfc\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.107894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107875 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.108040 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107929 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.108040 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.107960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.108040 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.108002 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-out\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.208637 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208605 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.208829 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208647 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.208829 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208674 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.208829 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208709 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.208829 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-web-config\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.208829 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:40.208761 2571 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Aug 04 16:00:40.208829 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:00:40.208831 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls podName:a8c16ae6-eb59-4b7f-9190-b6f5f6573428 nodeName:}" failed. No retries permitted until 2026-08-04 16:00:40.708815517 +0000 UTC m=+249.802415747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428") : secret "alertmanager-main-tls" not found Aug 04 16:00:40.209141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208766 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.209141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208958 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.209141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.208986 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhfc\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-kube-api-access-wlhfc\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.209141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.209036 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.209141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.209111 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.209428 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.209143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.209428 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.209228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-out\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.209428 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.209263 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-volume\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.213301 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.211747 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.213301 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.212155 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.213841 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.213681 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.213966 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.213940 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.214699 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.214634 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.214975 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.214952 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.215439 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.215415 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-web-config\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.216457 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.216431 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-volume\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.218228 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.217778 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.220059 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.220038 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-out\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.221016 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.220995 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.223305 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.223287 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhfc\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-kube-api-access-wlhfc\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.230374 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.230350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hnkg" event={"ID":"e6b15423-4ffc-4055-9c27-68928094fff7","Type":"ContainerStarted","Data":"96efa4ec94b6803b329c53523d77aff6ec36952474c10e352e835692cb399cfe"} Aug 04 16:00:40.718074 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.718030 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:40.718282 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.718130 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.721026 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.720998 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/aaa34a74-128c-489d-9427-eda3783de4bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6bb4d7b4dd-lbn9s\" (UID: \"aaa34a74-128c-489d-9427-eda3783de4bd\") " pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:40.721149 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.721047 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:40.833344 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.833312 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" Aug 04 16:00:40.942278 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.942242 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 16:00:40.987303 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:40.987227 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:00:41.145521 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.145481 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s"] Aug 04 16:00:41.148983 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:41.148955 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa34a74_128c_489d_9427_eda3783de4bd.slice/crio-1c006e3058112de4dc5ed465cc3aaa86f9fb265d0d637539fe4398c277f841b5 WatchSource:0}: Error finding container 1c006e3058112de4dc5ed465cc3aaa86f9fb265d0d637539fe4398c277f841b5: Status 404 returned error can't find the container with id 1c006e3058112de4dc5ed465cc3aaa86f9fb265d0d637539fe4398c277f841b5 Aug 04 16:00:41.171678 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.171650 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:00:41.176223 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:41.176182 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c16ae6_eb59_4b7f_9190_b6f5f6573428.slice/crio-fdb8a011cbbdba48a4c8a9a188c5bda021611eeaef10219fb85aa9232ee5a267 WatchSource:0}: Error finding container fdb8a011cbbdba48a4c8a9a188c5bda021611eeaef10219fb85aa9232ee5a267: Status 404 returned error can't find the container with id fdb8a011cbbdba48a4c8a9a188c5bda021611eeaef10219fb85aa9232ee5a267 Aug 04 16:00:41.235971 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.235944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerStarted","Data":"fdb8a011cbbdba48a4c8a9a188c5bda021611eeaef10219fb85aa9232ee5a267"} Aug 04 16:00:41.237689 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.237624 2571 generic.go:356] "Generic (PLEG): container finished" podID="e6b15423-4ffc-4055-9c27-68928094fff7" containerID="638bc9fc23d8efc51d154561f506a7e31e2b202be2d2c78e5f0543e62328e585" exitCode=0 Aug 04 16:00:41.237785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.237708 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hnkg" event={"ID":"e6b15423-4ffc-4055-9c27-68928094fff7","Type":"ContainerDied","Data":"638bc9fc23d8efc51d154561f506a7e31e2b202be2d2c78e5f0543e62328e585"} Aug 04 16:00:41.239758 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.239739 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" event={"ID":"aaa34a74-128c-489d-9427-eda3783de4bd","Type":"ContainerStarted","Data":"269c1e272d89fbac63db1624c63ab43a703bc3c0a293e020c935bf3e2ba2dd68"} Aug 04 16:00:41.239838 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.239789 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" event={"ID":"aaa34a74-128c-489d-9427-eda3783de4bd","Type":"ContainerStarted","Data":"1c006e3058112de4dc5ed465cc3aaa86f9fb265d0d637539fe4398c277f841b5"} Aug 04 16:00:41.287074 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.287045 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:41.383796 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.383770 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:41.544060 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:41.543973 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:42.044764 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.044604 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:42.044764 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.044705 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:42.050548 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.050517 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:42.250169 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.250131 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerID="7364b054451f90c5c40f458e4ee91f351e9da2da51b06a201c34084410c709cb" exitCode=0 Aug 04 16:00:42.250348 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.250247 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"7364b054451f90c5c40f458e4ee91f351e9da2da51b06a201c34084410c709cb"} Aug 04 16:00:42.253042 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.253011 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hnkg" event={"ID":"e6b15423-4ffc-4055-9c27-68928094fff7","Type":"ContainerStarted","Data":"383dac26b64ce89130292f0be03dada79ce6a9e8942cd4170c30c5d4dfa1fceb"} Aug 04 16:00:42.253151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.253087 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hnkg" event={"ID":"e6b15423-4ffc-4055-9c27-68928094fff7","Type":"ContainerStarted","Data":"4a8e3e7307e9ac90631b3e60a3dc9dd36c479ab5bd0aa67aead6eaf789564cca"} Aug 04 16:00:42.255272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.255246 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" event={"ID":"aaa34a74-128c-489d-9427-eda3783de4bd","Type":"ContainerStarted","Data":"6ccc37471452469ff46c3c42bbddb382457e925d69a2fe912f26529bc1c8c011"} Aug 04 16:00:42.261069 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.261047 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:00:42.335292 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.334770 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8hnkg" podStartSLOduration=2.193945938 podStartE2EDuration="3.334751079s" totalImagesPullingTime="1.072741861s" totalInitContainerRuntime="68.06328ms" isStatefulPod=true podCreationTimestamp="2026-08-04 16:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:42.333977421 +0000 UTC m=+251.427577669" watchObservedRunningTime="2026-08-04 16:00:42.334751079 +0000 UTC m=+251.428351379" Aug 04 16:00:42.854195 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:42.854163 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:43.262025 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.261982 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" event={"ID":"aaa34a74-128c-489d-9427-eda3783de4bd","Type":"ContainerStarted","Data":"2660c5e39f884daee3d353e3cdedd66b7d02e95b87bd7e7914edff103b588534"} Aug 04 16:00:43.284750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.284700 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6bb4d7b4dd-lbn9s" podStartSLOduration=3.949589461 podStartE2EDuration="5.284686548s" totalImagesPullingTime="1.335097087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:43.282323314 +0000 UTC m=+252.375923572" watchObservedRunningTime="2026-08-04 16:00:43.284686548 +0000 UTC m=+252.378286796" Aug 04 16:00:43.754684 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.754656 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-697h2"] Aug 04 16:00:43.758041 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.758022 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:43.761185 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.761164 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Aug 04 16:00:43.761327 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.761290 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rx8fk\"" Aug 04 16:00:43.777022 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.776987 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-697h2"] Aug 04 16:00:43.803881 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.803851 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64ddc68749-h65s2"] Aug 04 16:00:43.808351 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.808332 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.829907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.829883 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64ddc68749-h65s2"] Aug 04 16:00:43.848760 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.848654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73ee198a-29e5-43f2-80b4-92e28c7cc660-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-697h2\" (UID: \"73ee198a-29e5-43f2-80b4-92e28c7cc660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:43.848896 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.848762 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6kz\" (UniqueName: \"kubernetes.io/projected/73ee198a-29e5-43f2-80b4-92e28c7cc660-kube-api-access-px6kz\") pod \"monitoring-plugin-6dffbbd67-697h2\" (UID: \"73ee198a-29e5-43f2-80b4-92e28c7cc660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:43.949467 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949429 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6kz\" (UniqueName: \"kubernetes.io/projected/73ee198a-29e5-43f2-80b4-92e28c7cc660-kube-api-access-px6kz\") pod \"monitoring-plugin-6dffbbd67-697h2\" (UID: \"73ee198a-29e5-43f2-80b4-92e28c7cc660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:43.949664 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949481 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-trusted-ca-bundle\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.949664 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949524 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-oauth-serving-cert\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.949664 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949592 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-config\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.949808 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949725 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-service-ca\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.949808 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949762 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-oauth-config\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.949808 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949785 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s7qz\" (UniqueName: \"kubernetes.io/projected/0e16fb6a-3f9c-4a30-8c75-db46844568b4-kube-api-access-6s7qz\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.949923 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949810 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-serving-cert\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:43.949923 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.949843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73ee198a-29e5-43f2-80b4-92e28c7cc660-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-697h2\" (UID: \"73ee198a-29e5-43f2-80b4-92e28c7cc660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:43.952848 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.952823 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/73ee198a-29e5-43f2-80b4-92e28c7cc660-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-697h2\" (UID: \"73ee198a-29e5-43f2-80b4-92e28c7cc660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:43.959162 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:43.959134 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6kz\" (UniqueName: \"kubernetes.io/projected/73ee198a-29e5-43f2-80b4-92e28c7cc660-kube-api-access-px6kz\") pod \"monitoring-plugin-6dffbbd67-697h2\" (UID: \"73ee198a-29e5-43f2-80b4-92e28c7cc660\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:44.050558 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.050484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-trusted-ca-bundle\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.050558 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.050525 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-oauth-serving-cert\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.050558 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.050558 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-config\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.050791 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.050660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-service-ca\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.050791 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.050696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-oauth-config\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.050791 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.050719 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s7qz\" (UniqueName: \"kubernetes.io/projected/0e16fb6a-3f9c-4a30-8c75-db46844568b4-kube-api-access-6s7qz\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.050947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.050836 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-serving-cert\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.051419 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.051386 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-oauth-serving-cert\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.051419 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.051409 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-config\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.051576 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.051450 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-trusted-ca-bundle\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.051576 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.051512 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-service-ca\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.053450 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.053422 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-oauth-config\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.053564 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.053547 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-serving-cert\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.062751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.062725 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s7qz\" (UniqueName: \"kubernetes.io/projected/0e16fb6a-3f9c-4a30-8c75-db46844568b4-kube-api-access-6s7qz\") pod \"console-64ddc68749-h65s2\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.067412 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.067391 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:44.118307 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.118276 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:44.177548 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.177434 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:44.240649 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.240348 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-697h2"] Aug 04 16:00:44.243469 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:44.243411 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ee198a_29e5_43f2_80b4_92e28c7cc660.slice/crio-e32a05ad31f3525da694676225759a7cea3e7cc12f148d61bd163ae33ebf2c75 WatchSource:0}: Error finding container e32a05ad31f3525da694676225759a7cea3e7cc12f148d61bd163ae33ebf2c75: Status 404 returned error can't find the container with id e32a05ad31f3525da694676225759a7cea3e7cc12f148d61bd163ae33ebf2c75 Aug 04 16:00:44.243604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.243503 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc"] Aug 04 16:00:44.247547 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.247505 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.251800 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.251529 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Aug 04 16:00:44.251800 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.251753 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nl4pb\"" Aug 04 16:00:44.251954 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.251889 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Aug 04 16:00:44.252560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.252051 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Aug 04 16:00:44.252560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.252276 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Aug 04 16:00:44.252560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.252529 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Aug 04 16:00:44.261073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.261047 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Aug 04 16:00:44.271663 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.269781 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc"] Aug 04 16:00:44.275239 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.273161 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:44.283416 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.283383 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" event={"ID":"73ee198a-29e5-43f2-80b4-92e28c7cc660","Type":"ContainerStarted","Data":"e32a05ad31f3525da694676225759a7cea3e7cc12f148d61bd163ae33ebf2c75"} Aug 04 16:00:44.285878 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.285803 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerStarted","Data":"19a4dfcb94e0dd595f4e13d9add4e2326ef4f877e0946199d9d298d9d29af652"} Aug 04 16:00:44.336191 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.336167 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64ddc68749-h65s2"] Aug 04 16:00:44.339088 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:44.339061 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e16fb6a_3f9c_4a30_8c75_db46844568b4.slice/crio-1f2b718422769344bb181b629c4fb6633279a302ccfbeef191a88b89381cdb85 WatchSource:0}: Error finding container 1f2b718422769344bb181b629c4fb6633279a302ccfbeef191a88b89381cdb85: Status 404 returned error can't find the container with id 1f2b718422769344bb181b629c4fb6633279a302ccfbeef191a88b89381cdb85 Aug 04 16:00:44.354148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.354122 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-telemeter-client-tls\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.354340 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.354313 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-federate-client-tls\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.354433 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.354367 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.354495 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.354447 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjh8\" (UniqueName: \"kubernetes.io/projected/b692cf55-5d89-4447-81a9-e4f426962d47-kube-api-access-hfjh8\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.354665 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.354640 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-metrics-client-ca\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.354770 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.354731 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-serving-certs-ca-bundle\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.354977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.354949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-secret-telemeter-client\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.355337 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.355197 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.456578 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.456690 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456624 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-telemeter-client-tls\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.456690 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456663 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-federate-client-tls\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.457014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.457014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456736 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjh8\" (UniqueName: \"kubernetes.io/projected/b692cf55-5d89-4447-81a9-e4f426962d47-kube-api-access-hfjh8\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.457014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-metrics-client-ca\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.457014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456827 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-serving-certs-ca-bundle\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.457014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.456850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-secret-telemeter-client\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.457565 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.457540 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.458166 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.458128 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-serving-certs-ca-bundle\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.458421 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.458398 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b692cf55-5d89-4447-81a9-e4f426962d47-metrics-client-ca\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.460194 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.460134 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.460316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.460299 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-federate-client-tls\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.460427 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.460406 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-telemeter-client-tls\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.460563 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.460545 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b692cf55-5d89-4447-81a9-e4f426962d47-secret-telemeter-client\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.467053 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.467026 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjh8\" (UniqueName: \"kubernetes.io/projected/b692cf55-5d89-4447-81a9-e4f426962d47-kube-api-access-hfjh8\") pod \"telemeter-client-6cb56c5f9c-c6khc\" (UID: \"b692cf55-5d89-4447-81a9-e4f426962d47\") " pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.518595 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.518564 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:44.559872 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.559844 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" Aug 04 16:00:44.641878 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.641839 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:44.695281 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:44.695247 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc"] Aug 04 16:00:44.698156 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:00:44.698132 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb692cf55_5d89_4447_81a9_e4f426962d47.slice/crio-c95897002a8275c8879e7bf7aac3a3903a8345f8a51b847c43732820bc14ce7d WatchSource:0}: Error finding container c95897002a8275c8879e7bf7aac3a3903a8345f8a51b847c43732820bc14ce7d: Status 404 returned error can't find the container with id c95897002a8275c8879e7bf7aac3a3903a8345f8a51b847c43732820bc14ce7d Aug 04 16:00:45.261527 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.261482 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:45.290715 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.290681 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64ddc68749-h65s2" event={"ID":"0e16fb6a-3f9c-4a30-8c75-db46844568b4","Type":"ContainerStarted","Data":"fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5"} Aug 04 16:00:45.291168 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.290733 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64ddc68749-h65s2" event={"ID":"0e16fb6a-3f9c-4a30-8c75-db46844568b4","Type":"ContainerStarted","Data":"1f2b718422769344bb181b629c4fb6633279a302ccfbeef191a88b89381cdb85"} Aug 04 16:00:45.291946 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.291919 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" event={"ID":"b692cf55-5d89-4447-81a9-e4f426962d47","Type":"ContainerStarted","Data":"c95897002a8275c8879e7bf7aac3a3903a8345f8a51b847c43732820bc14ce7d"} Aug 04 16:00:45.294673 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.294654 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerStarted","Data":"f180e0ac51ac54e11125e6184c4bbfe3d2fc437c0ab2cfdd51649de110f96ee4"} Aug 04 16:00:45.294743 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.294702 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerStarted","Data":"71df0d2e5c6e8c9884f32b5943bee8cff582f9690dfc841607fb6d503f7e9022"} Aug 04 16:00:45.294743 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.294728 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerStarted","Data":"3a7501250e34b2f9ae871bbbd11d342057f6d18a82b87b9af04d75d98eec5561"} Aug 04 16:00:45.294821 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.294751 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerStarted","Data":"834a23d3682d11010fd5e0865ddb181a342af3fc2388cfee5131bc4de5daa9df"} Aug 04 16:00:45.315055 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.315015 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64ddc68749-h65s2" podStartSLOduration=2.315001486 podStartE2EDuration="2.315001486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:45.314650776 +0000 UTC m=+254.408251023" watchObservedRunningTime="2026-08-04 16:00:45.315001486 +0000 UTC m=+254.408601731" Aug 04 16:00:45.954112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:45.954062 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" containerID="cri-o://527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284" gracePeriod=30 Aug 04 16:00:46.302712 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:46.302619 2571 generic.go:356] "Generic (PLEG): container finished" podID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerID="527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284" exitCode=0 Aug 04 16:00:46.303148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:46.302713 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerDied","Data":"527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284"} Aug 04 16:00:46.303148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:46.302780 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerStarted","Data":"a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b"} Aug 04 16:00:46.303148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:46.302814 2571 scope.go:117] "RemoveContainer" containerID="df9347d63e752d9e26d159390b99a7b1811df1dc925208399346d8cd7bb71a3d" Aug 04 16:00:46.303148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:46.302966 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 16:00:47.311105 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:47.311062 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerStarted","Data":"d3f34c278ce09405c535410b935a9e4a7f6d8e038cf0a275f4530a5c455ede63"} Aug 04 16:00:47.343350 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:47.343183 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.324328584 podStartE2EDuration="7.343161217s" totalImagesPullingTime="5.003965291s" totalInitContainerRuntime="14.867342ms" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:47.342448711 +0000 UTC m=+256.436048972" watchObservedRunningTime="2026-08-04 16:00:47.343161217 +0000 UTC m=+256.436761465" Aug 04 16:00:47.379261 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:47.379198 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:47.478352 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:47.478322 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:48.319068 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.319035 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" event={"ID":"73ee198a-29e5-43f2-80b4-92e28c7cc660","Type":"ContainerStarted","Data":"afef6ea8dc16234dd88c7bdf9ca0ece5a6d79fa2034696ddf81ea88a366ab4f0"} Aug 04 16:00:48.319472 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.319235 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:48.321458 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.321434 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" event={"ID":"b692cf55-5d89-4447-81a9-e4f426962d47","Type":"ContainerStarted","Data":"38707c0291f33dca2ff28461f8f9a3656434959ca7ace81e8959a8fa4d436d04"} Aug 04 16:00:48.321558 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.321491 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" event={"ID":"b692cf55-5d89-4447-81a9-e4f426962d47","Type":"ContainerStarted","Data":"a3cfadad15a3a9eae224a6254b3e21cc850c6f561e6b218e0ea0f2130dc2c79b"} Aug 04 16:00:48.321558 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.321517 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" event={"ID":"b692cf55-5d89-4447-81a9-e4f426962d47","Type":"ContainerStarted","Data":"ec058dce1c6219f9565dbe3f56e744f5e9307fd0c44943eb40db48f8215b78a3"} Aug 04 16:00:48.324785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.324770 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" Aug 04 16:00:48.337185 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.337147 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-697h2" podStartSLOduration=1.349782674 podStartE2EDuration="5.337135241s" totalImagesPullingTime="3.987352567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:48.336184786 +0000 UTC m=+257.429785033" watchObservedRunningTime="2026-08-04 16:00:48.337135241 +0000 UTC m=+257.430735488" Aug 04 16:00:48.381448 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.381339 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cb56c5f9c-c6khc" podStartSLOduration=1.661533581 podStartE2EDuration="4.381322647s" totalImagesPullingTime="2.719789066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:00:48.379627263 +0000 UTC m=+257.473227511" watchObservedRunningTime="2026-08-04 16:00:48.381322647 +0000 UTC m=+257.474922896" Aug 04 16:00:48.476456 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:48.476416 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64ddc68749-h65s2"] Aug 04 16:00:54.118673 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:54.118594 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:00:54.204774 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:54.204744 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69558cbb7-9j4hh" Aug 04 16:00:54.394589 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:54.394500 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:54.477762 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:54.477724 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:55.406599 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:55.406564 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:55.507526 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:55.507494 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:59.093557 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:59.093525 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854ddc5575-ng4bw"] Aug 04 16:00:59.255906 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:59.255868 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:59.437324 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:59.437224 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:59.539171 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:00:59.539141 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:06.308265 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:06.308236 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 16:01:11.326370 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.326322 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" containerID="cri-o://a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b" gracePeriod=30 Aug 04 16:01:11.574847 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.574821 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 16:01:11.599763 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599670 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-trusted-ca\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.599763 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599729 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n2tp\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-kube-api-access-9n2tp\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.599977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599775 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-certificates\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.599977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599822 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/023f756f-dd53-42b9-a4ab-7411cb56ecef-ca-trust-extracted\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.599977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599866 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-installation-pull-secrets\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.599977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599899 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-image-registry-private-configuration\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.599977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599960 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-bound-sa-token\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.600148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.599997 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") pod \"023f756f-dd53-42b9-a4ab-7411cb56ecef\" (UID: \"023f756f-dd53-42b9-a4ab-7411cb56ecef\") " Aug 04 16:01:11.600148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.600070 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:11.600322 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.600302 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-trusted-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:11.602481 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.602419 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:11.603509 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.603417 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:01:11.603664 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.603469 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:01:11.603664 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.603621 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-kube-api-access-9n2tp" (OuterVolumeSpecName: "kube-api-access-9n2tp") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "kube-api-access-9n2tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:01:11.604069 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.604047 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:01:11.604255 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.604146 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:01:11.611304 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.611271 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023f756f-dd53-42b9-a4ab-7411cb56ecef-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "023f756f-dd53-42b9-a4ab-7411cb56ecef" (UID: "023f756f-dd53-42b9-a4ab-7411cb56ecef"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:01:11.701445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.701413 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-bound-sa-token\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:11.701445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.701441 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:11.701445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.701450 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n2tp\" (UniqueName: \"kubernetes.io/projected/023f756f-dd53-42b9-a4ab-7411cb56ecef-kube-api-access-9n2tp\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:11.701659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.701460 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/023f756f-dd53-42b9-a4ab-7411cb56ecef-registry-certificates\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:11.701659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.701470 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/023f756f-dd53-42b9-a4ab-7411cb56ecef-ca-trust-extracted\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:11.701659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.701478 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-installation-pull-secrets\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:11.701659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:11.701488 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/023f756f-dd53-42b9-a4ab-7411cb56ecef-image-registry-private-configuration\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:12.412807 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.412773 2571 generic.go:356] "Generic (PLEG): container finished" podID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerID="a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b" exitCode=0 Aug 04 16:01:12.413184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.412822 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" Aug 04 16:01:12.413184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.412819 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerDied","Data":"a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b"} Aug 04 16:01:12.413184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.412946 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7994b49464-qzkr9" event={"ID":"023f756f-dd53-42b9-a4ab-7411cb56ecef","Type":"ContainerDied","Data":"1081396a0b0877c238d8a3be4de79d23e9201891cc384aeb0bbe01b55de98205"} Aug 04 16:01:12.413184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.412978 2571 scope.go:117] "RemoveContainer" containerID="a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b" Aug 04 16:01:12.421698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.421675 2571 scope.go:117] "RemoveContainer" containerID="527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284" Aug 04 16:01:12.429013 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.428994 2571 scope.go:117] "RemoveContainer" containerID="a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b" Aug 04 16:01:12.429347 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:01:12.429325 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b\": container with ID starting with a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b not found: ID does not exist" containerID="a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b" Aug 04 16:01:12.429411 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.429355 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b"} err="failed to get container status \"a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b\": rpc error: code = NotFound desc = could not find container \"a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b\": container with ID starting with a4e1357e006aea405448dacdb3c158e1d86599e1e0a51a4eaeb9fb8fe3f7a73b not found: ID does not exist" Aug 04 16:01:12.429411 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.429373 2571 scope.go:117] "RemoveContainer" containerID="527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284" Aug 04 16:01:12.429599 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:01:12.429580 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284\": container with ID starting with 527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284 not found: ID does not exist" containerID="527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284" Aug 04 16:01:12.429652 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.429610 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284"} err="failed to get container status \"527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284\": rpc error: code = NotFound desc = could not find container \"527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284\": container with ID starting with 527e03a98ffa0890da2d5b1ce422d820b66d47e9bbf1e52cde2fdaa63c700284 not found: ID does not exist" Aug 04 16:01:12.434770 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.434747 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7994b49464-qzkr9"] Aug 04 16:01:12.439330 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:12.439309 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7994b49464-qzkr9"] Aug 04 16:01:13.362016 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.361982 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" path="/var/lib/kubelet/pods/023f756f-dd53-42b9-a4ab-7411cb56ecef/volumes" Aug 04 16:01:13.495996 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.495959 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64ddc68749-h65s2" podUID="0e16fb6a-3f9c-4a30-8c75-db46844568b4" containerName="console" containerID="cri-o://fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5" gracePeriod=15 Aug 04 16:01:13.731622 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.731601 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:01:13.820135 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820106 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-serving-cert\") pod \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " Aug 04 16:01:13.820305 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820154 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-trusted-ca-bundle\") pod \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " Aug 04 16:01:13.820305 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820190 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-config\") pod \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " Aug 04 16:01:13.820305 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820233 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-oauth-config\") pod \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " Aug 04 16:01:13.820453 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820354 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-oauth-serving-cert\") pod \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " Aug 04 16:01:13.820513 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820456 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s7qz\" (UniqueName: \"kubernetes.io/projected/0e16fb6a-3f9c-4a30-8c75-db46844568b4-kube-api-access-6s7qz\") pod \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " Aug 04 16:01:13.820513 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820499 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-service-ca\") pod \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\" (UID: \"0e16fb6a-3f9c-4a30-8c75-db46844568b4\") " Aug 04 16:01:13.820631 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820611 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-config" (OuterVolumeSpecName: "console-config") pod "0e16fb6a-3f9c-4a30-8c75-db46844568b4" (UID: "0e16fb6a-3f9c-4a30-8c75-db46844568b4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:13.820689 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820646 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0e16fb6a-3f9c-4a30-8c75-db46844568b4" (UID: "0e16fb6a-3f9c-4a30-8c75-db46844568b4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:13.820744 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820722 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0e16fb6a-3f9c-4a30-8c75-db46844568b4" (UID: "0e16fb6a-3f9c-4a30-8c75-db46844568b4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:13.820893 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820855 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:13.820893 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820883 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:13.821131 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.820927 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-service-ca" (OuterVolumeSpecName: "service-ca") pod "0e16fb6a-3f9c-4a30-8c75-db46844568b4" (UID: "0e16fb6a-3f9c-4a30-8c75-db46844568b4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:13.822355 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.822339 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0e16fb6a-3f9c-4a30-8c75-db46844568b4" (UID: "0e16fb6a-3f9c-4a30-8c75-db46844568b4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:01:13.822453 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.822439 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0e16fb6a-3f9c-4a30-8c75-db46844568b4" (UID: "0e16fb6a-3f9c-4a30-8c75-db46844568b4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:01:13.822493 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.822467 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e16fb6a-3f9c-4a30-8c75-db46844568b4-kube-api-access-6s7qz" (OuterVolumeSpecName: "kube-api-access-6s7qz") pod "0e16fb6a-3f9c-4a30-8c75-db46844568b4" (UID: "0e16fb6a-3f9c-4a30-8c75-db46844568b4"). InnerVolumeSpecName "kube-api-access-6s7qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:01:13.922101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.922017 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s7qz\" (UniqueName: \"kubernetes.io/projected/0e16fb6a-3f9c-4a30-8c75-db46844568b4-kube-api-access-6s7qz\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:13.922101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.922057 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:13.922101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.922067 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:13.922101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.922077 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e16fb6a-3f9c-4a30-8c75-db46844568b4-console-oauth-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:13.922101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:13.922085 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e16fb6a-3f9c-4a30-8c75-db46844568b4-oauth-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:14.422540 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.422503 2571 generic.go:356] "Generic (PLEG): container finished" podID="0e16fb6a-3f9c-4a30-8c75-db46844568b4" containerID="fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5" exitCode=0 Aug 04 16:01:14.422702 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.422545 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64ddc68749-h65s2" Aug 04 16:01:14.422702 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.422545 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64ddc68749-h65s2" event={"ID":"0e16fb6a-3f9c-4a30-8c75-db46844568b4","Type":"ContainerDied","Data":"fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5"} Aug 04 16:01:14.422702 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.422599 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64ddc68749-h65s2" event={"ID":"0e16fb6a-3f9c-4a30-8c75-db46844568b4","Type":"ContainerDied","Data":"1f2b718422769344bb181b629c4fb6633279a302ccfbeef191a88b89381cdb85"} Aug 04 16:01:14.422702 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.422626 2571 scope.go:117] "RemoveContainer" containerID="fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5" Aug 04 16:01:14.432065 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.432049 2571 scope.go:117] "RemoveContainer" containerID="fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5" Aug 04 16:01:14.432314 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:01:14.432297 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5\": container with ID starting with fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5 not found: ID does not exist" containerID="fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5" Aug 04 16:01:14.432383 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.432321 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5"} err="failed to get container status \"fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5\": rpc error: code = NotFound desc = could not find container \"fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5\": container with ID starting with fa07365d6b3dd66f13c097e87711d6168f60d5ded9bc8bf14759c9a01f7b3ed5 not found: ID does not exist" Aug 04 16:01:14.447005 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.446983 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64ddc68749-h65s2"] Aug 04 16:01:14.452834 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:14.452817 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64ddc68749-h65s2"] Aug 04 16:01:15.362465 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:15.362429 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e16fb6a-3f9c-4a30-8c75-db46844568b4" path="/var/lib/kubelet/pods/0e16fb6a-3f9c-4a30-8c75-db46844568b4/volumes" Aug 04 16:01:16.897534 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:16.897491 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:16.971328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:16.971298 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:24.115199 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.115145 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-854ddc5575-ng4bw" podUID="34309738-5c9b-44c1-a926-207b530e8440" containerName="console" containerID="cri-o://e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017" gracePeriod=15 Aug 04 16:01:24.353888 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.353857 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:01:24.402406 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402309 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc6pw\" (UniqueName: \"kubernetes.io/projected/34309738-5c9b-44c1-a926-207b530e8440-kube-api-access-gc6pw\") pod \"34309738-5c9b-44c1-a926-207b530e8440\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " Aug 04 16:01:24.402406 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402387 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-console-config\") pod \"34309738-5c9b-44c1-a926-207b530e8440\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " Aug 04 16:01:24.402643 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402415 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-oauth-config\") pod \"34309738-5c9b-44c1-a926-207b530e8440\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " Aug 04 16:01:24.402643 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402432 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-oauth-serving-cert\") pod \"34309738-5c9b-44c1-a926-207b530e8440\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " Aug 04 16:01:24.402643 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402458 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-service-ca\") pod \"34309738-5c9b-44c1-a926-207b530e8440\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " Aug 04 16:01:24.402643 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402495 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-serving-cert\") pod \"34309738-5c9b-44c1-a926-207b530e8440\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " Aug 04 16:01:24.402643 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402525 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-trusted-ca-bundle\") pod \"34309738-5c9b-44c1-a926-207b530e8440\" (UID: \"34309738-5c9b-44c1-a926-207b530e8440\") " Aug 04 16:01:24.403000 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402868 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "34309738-5c9b-44c1-a926-207b530e8440" (UID: "34309738-5c9b-44c1-a926-207b530e8440"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:24.403075 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402974 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-service-ca" (OuterVolumeSpecName: "service-ca") pod "34309738-5c9b-44c1-a926-207b530e8440" (UID: "34309738-5c9b-44c1-a926-207b530e8440"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:24.403075 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.402985 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-console-config" (OuterVolumeSpecName: "console-config") pod "34309738-5c9b-44c1-a926-207b530e8440" (UID: "34309738-5c9b-44c1-a926-207b530e8440"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:24.403075 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.403064 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "34309738-5c9b-44c1-a926-207b530e8440" (UID: "34309738-5c9b-44c1-a926-207b530e8440"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:01:24.404656 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.404618 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "34309738-5c9b-44c1-a926-207b530e8440" (UID: "34309738-5c9b-44c1-a926-207b530e8440"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:01:24.404927 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.404891 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "34309738-5c9b-44c1-a926-207b530e8440" (UID: "34309738-5c9b-44c1-a926-207b530e8440"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:01:24.405032 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.405010 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34309738-5c9b-44c1-a926-207b530e8440-kube-api-access-gc6pw" (OuterVolumeSpecName: "kube-api-access-gc6pw") pod "34309738-5c9b-44c1-a926-207b530e8440" (UID: "34309738-5c9b-44c1-a926-207b530e8440"). InnerVolumeSpecName "kube-api-access-gc6pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:01:24.465891 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.465852 2571 generic.go:356] "Generic (PLEG): container finished" podID="34309738-5c9b-44c1-a926-207b530e8440" containerID="e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017" exitCode=0 Aug 04 16:01:24.466085 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.465898 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-854ddc5575-ng4bw" Aug 04 16:01:24.466085 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.465924 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854ddc5575-ng4bw" event={"ID":"34309738-5c9b-44c1-a926-207b530e8440","Type":"ContainerDied","Data":"e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017"} Aug 04 16:01:24.466085 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.465979 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-854ddc5575-ng4bw" event={"ID":"34309738-5c9b-44c1-a926-207b530e8440","Type":"ContainerDied","Data":"63ed454f5f397dadaec99635d15bcce3b95b06991d0e2287661d4a8cc6420427"} Aug 04 16:01:24.466085 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.466009 2571 scope.go:117] "RemoveContainer" containerID="e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017" Aug 04 16:01:24.474578 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.474559 2571 scope.go:117] "RemoveContainer" containerID="e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017" Aug 04 16:01:24.474888 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:01:24.474866 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017\": container with ID starting with e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017 not found: ID does not exist" containerID="e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017" Aug 04 16:01:24.474941 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.474898 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017"} err="failed to get container status \"e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017\": rpc error: code = NotFound desc = could not find container \"e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017\": container with ID starting with e0794ce188ee05fac2a7967e769b08fe499af0f718207dca5de252bb930bb017 not found: ID does not exist" Aug 04 16:01:24.487758 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.487725 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-854ddc5575-ng4bw"] Aug 04 16:01:24.492539 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.492508 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-854ddc5575-ng4bw"] Aug 04 16:01:24.503388 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.503351 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc6pw\" (UniqueName: \"kubernetes.io/projected/34309738-5c9b-44c1-a926-207b530e8440-kube-api-access-gc6pw\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:24.503388 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.503385 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-console-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:24.503388 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.503398 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-oauth-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:24.503602 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.503407 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-oauth-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:24.503602 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.503416 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:24.503602 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.503425 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/34309738-5c9b-44c1-a926-207b530e8440-console-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:24.503602 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:24.503433 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34309738-5c9b-44c1-a926-207b530e8440-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:01:25.361841 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:25.361811 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34309738-5c9b-44c1-a926-207b530e8440" path="/var/lib/kubelet/pods/34309738-5c9b-44c1-a926-207b530e8440/volumes" Aug 04 16:01:27.878698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:27.878646 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:30.423296 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:30.423258 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:30.487650 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:30.487615 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:31.285659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:31.285365 2571 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 16:01:32.876224 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:32.876170 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:56.143670 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:56.143631 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:01:58.419228 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419177 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6967b9f947-qkdkm"] Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419472 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419483 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419495 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419502 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419510 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34309738-5c9b-44c1-a926-207b530e8440" containerName="console" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419517 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="34309738-5c9b-44c1-a926-207b530e8440" containerName="console" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419529 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419534 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419545 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e16fb6a-3f9c-4a30-8c75-db46844568b4" containerName="console" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419550 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e16fb6a-3f9c-4a30-8c75-db46844568b4" containerName="console" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419596 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419602 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="34309738-5c9b-44c1-a926-207b530e8440" containerName="console" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419608 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e16fb6a-3f9c-4a30-8c75-db46844568b4" containerName="console" Aug 04 16:01:58.419625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.419615 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:01:58.421668 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.421649 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.426234 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.426181 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Aug 04 16:01:58.426431 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.426405 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Aug 04 16:01:58.426552 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.426462 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Aug 04 16:01:58.426552 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.426496 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Aug 04 16:01:58.426552 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.426506 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Aug 04 16:01:58.427945 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.427917 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Aug 04 16:01:58.428061 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.427969 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Aug 04 16:01:58.428061 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.428030 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vmh6q\"" Aug 04 16:01:58.431855 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.431837 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Aug 04 16:01:58.440185 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.440162 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6967b9f947-qkdkm"] Aug 04 16:01:58.491871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.491840 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-console-config\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.491871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.491873 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-oauth-serving-cert\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.492087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.491895 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-serving-cert\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.492087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.491915 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-oauth-config\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.492087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.491958 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-service-ca\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.492087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.492068 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcth\" (UniqueName: \"kubernetes.io/projected/345e2a49-35f6-4433-8969-7cb7b593add3-kube-api-access-hqcth\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.492235 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.492099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-trusted-ca-bundle\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.592506 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.592466 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-console-config\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.592698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.592516 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-oauth-serving-cert\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.592698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.592548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-serving-cert\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.592698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.592586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-oauth-config\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.592698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.592606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-service-ca\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.592698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.592679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcth\" (UniqueName: \"kubernetes.io/projected/345e2a49-35f6-4433-8969-7cb7b593add3-kube-api-access-hqcth\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.592969 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.592705 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-trusted-ca-bundle\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.593359 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.593308 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-console-config\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.593592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.593560 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-oauth-serving-cert\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.593704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.593660 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-trusted-ca-bundle\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.593908 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.593887 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-service-ca\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.595544 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.595516 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-serving-cert\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.595647 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.595569 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-oauth-config\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.604477 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.604436 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcth\" (UniqueName: \"kubernetes.io/projected/345e2a49-35f6-4433-8969-7cb7b593add3-kube-api-access-hqcth\") pod \"console-6967b9f947-qkdkm\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.731981 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.731888 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:01:58.862082 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:58.862055 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6967b9f947-qkdkm"] Aug 04 16:01:58.864312 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:01:58.864282 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod345e2a49_35f6_4433_8969_7cb7b593add3.slice/crio-4391d406dea451850dccf13b177f999690d9998c9fa56e75ffd4399957348ba4 WatchSource:0}: Error finding container 4391d406dea451850dccf13b177f999690d9998c9fa56e75ffd4399957348ba4: Status 404 returned error can't find the container with id 4391d406dea451850dccf13b177f999690d9998c9fa56e75ffd4399957348ba4 Aug 04 16:01:59.381768 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.381733 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:01:59.382058 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.382035 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="alertmanager" containerID="cri-o://19a4dfcb94e0dd595f4e13d9add4e2326ef4f877e0946199d9d298d9d29af652" gracePeriod=120 Aug 04 16:01:59.382142 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.382110 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-metric" containerID="cri-o://f180e0ac51ac54e11125e6184c4bbfe3d2fc437c0ab2cfdd51649de110f96ee4" gracePeriod=120 Aug 04 16:01:59.382254 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.382124 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="prom-label-proxy" containerID="cri-o://d3f34c278ce09405c535410b935a9e4a7f6d8e038cf0a275f4530a5c455ede63" gracePeriod=120 Aug 04 16:01:59.382254 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.382184 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="config-reloader" containerID="cri-o://834a23d3682d11010fd5e0865ddb181a342af3fc2388cfee5131bc4de5daa9df" gracePeriod=120 Aug 04 16:01:59.382254 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.382140 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-web" containerID="cri-o://3a7501250e34b2f9ae871bbbd11d342057f6d18a82b87b9af04d75d98eec5561" gracePeriod=120 Aug 04 16:01:59.382401 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.382191 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy" containerID="cri-o://71df0d2e5c6e8c9884f32b5943bee8cff582f9690dfc841607fb6d503f7e9022" gracePeriod=120 Aug 04 16:01:59.607909 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.607872 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967b9f947-qkdkm" event={"ID":"345e2a49-35f6-4433-8969-7cb7b593add3","Type":"ContainerStarted","Data":"5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10"} Aug 04 16:01:59.608360 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.607963 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967b9f947-qkdkm" event={"ID":"345e2a49-35f6-4433-8969-7cb7b593add3","Type":"ContainerStarted","Data":"4391d406dea451850dccf13b177f999690d9998c9fa56e75ffd4399957348ba4"} Aug 04 16:01:59.611145 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611120 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerID="d3f34c278ce09405c535410b935a9e4a7f6d8e038cf0a275f4530a5c455ede63" exitCode=0 Aug 04 16:01:59.611145 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611144 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerID="71df0d2e5c6e8c9884f32b5943bee8cff582f9690dfc841607fb6d503f7e9022" exitCode=0 Aug 04 16:01:59.611145 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611150 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerID="834a23d3682d11010fd5e0865ddb181a342af3fc2388cfee5131bc4de5daa9df" exitCode=0 Aug 04 16:01:59.611386 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611156 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerID="19a4dfcb94e0dd595f4e13d9add4e2326ef4f877e0946199d9d298d9d29af652" exitCode=0 Aug 04 16:01:59.611386 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611189 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"d3f34c278ce09405c535410b935a9e4a7f6d8e038cf0a275f4530a5c455ede63"} Aug 04 16:01:59.611386 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611277 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"71df0d2e5c6e8c9884f32b5943bee8cff582f9690dfc841607fb6d503f7e9022"} Aug 04 16:01:59.611386 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611323 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"834a23d3682d11010fd5e0865ddb181a342af3fc2388cfee5131bc4de5daa9df"} Aug 04 16:01:59.611386 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.611363 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"19a4dfcb94e0dd595f4e13d9add4e2326ef4f877e0946199d9d298d9d29af652"} Aug 04 16:01:59.628415 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:01:59.628360 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6967b9f947-qkdkm" podStartSLOduration=1.628319576 podStartE2EDuration="1.628319576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:01:59.627030207 +0000 UTC m=+328.720630451" watchObservedRunningTime="2026-08-04 16:01:59.628319576 +0000 UTC m=+328.721919823" Aug 04 16:02:00.618524 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.618493 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerID="f180e0ac51ac54e11125e6184c4bbfe3d2fc437c0ab2cfdd51649de110f96ee4" exitCode=0 Aug 04 16:02:00.618524 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.618517 2571 generic.go:356] "Generic (PLEG): container finished" podID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerID="3a7501250e34b2f9ae871bbbd11d342057f6d18a82b87b9af04d75d98eec5561" exitCode=0 Aug 04 16:02:00.618932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.618535 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"f180e0ac51ac54e11125e6184c4bbfe3d2fc437c0ab2cfdd51649de110f96ee4"} Aug 04 16:02:00.618932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.618605 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"3a7501250e34b2f9ae871bbbd11d342057f6d18a82b87b9af04d75d98eec5561"} Aug 04 16:02:00.618932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.618640 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a8c16ae6-eb59-4b7f-9190-b6f5f6573428","Type":"ContainerDied","Data":"fdb8a011cbbdba48a4c8a9a188c5bda021611eeaef10219fb85aa9232ee5a267"} Aug 04 16:02:00.618932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.618665 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdb8a011cbbdba48a4c8a9a188c5bda021611eeaef10219fb85aa9232ee5a267" Aug 04 16:02:00.620670 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.620653 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:00.713273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713172 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-web-config\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713224 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-trusted-ca-bundle\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713244 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713267 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713293 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-tls-assets\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713316 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-web\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713335 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-out\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713361 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-volume\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713389 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713430 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlhfc\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-kube-api-access-wlhfc\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713506 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-cluster-tls-config\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713537 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-main-db\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.713587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713566 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-metrics-client-ca\") pod \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\" (UID: \"a8c16ae6-eb59-4b7f-9190-b6f5f6573428\") " Aug 04 16:02:00.714039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713602 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:02:00.714039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.713933 2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.714551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.714524 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:02:00.714871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.714845 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:02:00.716171 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.715918 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:00.716171 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.715939 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:02:00.716171 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.716054 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:00.716615 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.716583 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:00.716838 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.716808 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:00.717969 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.717947 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-out" (OuterVolumeSpecName: "config-out") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:02:00.718058 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.718036 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-kube-api-access-wlhfc" (OuterVolumeSpecName: "kube-api-access-wlhfc") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "kube-api-access-wlhfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:02:00.718535 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.718509 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-volume" (OuterVolumeSpecName: "config-volume") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:00.721467 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.721151 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:00.726907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.726883 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-web-config" (OuterVolumeSpecName: "web-config") pod "a8c16ae6-eb59-4b7f-9190-b6f5f6573428" (UID: "a8c16ae6-eb59-4b7f-9190-b6f5f6573428"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:02:00.815262 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815226 2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-web-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815262 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815257 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815262 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815267 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815278 2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-tls-assets\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815287 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815295 2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-out\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815303 2571 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-config-volume\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815311 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-secret-alertmanager-main-tls\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815319 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlhfc\" (UniqueName: \"kubernetes.io/projected/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-kube-api-access-wlhfc\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815327 2571 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-cluster-tls-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815334 2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-alertmanager-main-db\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:00.815530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:00.815342 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8c16ae6-eb59-4b7f-9190-b6f5f6573428-metrics-client-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:02:01.622080 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.621995 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.658682 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.658646 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:02:01.666634 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.666604 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:02:01.701256 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701226 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:02:01.701516 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701504 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:02:01.701559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701518 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:02:01.701559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701529 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy" Aug 04 16:02:01.701559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701536 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy" Aug 04 16:02:01.701559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701542 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="alertmanager" Aug 04 16:02:01.701559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701548 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="alertmanager" Aug 04 16:02:01.701559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701561 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="config-reloader" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701566 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="config-reloader" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701572 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="prom-label-proxy" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701578 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="prom-label-proxy" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701586 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-web" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701591 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-web" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701598 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-metric" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701602 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-metric" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701614 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="init-config-reloader" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701620 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="init-config-reloader" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701669 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701677 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="prom-label-proxy" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701684 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="config-reloader" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701691 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="alertmanager" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701697 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-metric" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701703 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy" Aug 04 16:02:01.701728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701708 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" containerName="kube-rbac-proxy-web" Aug 04 16:02:01.702174 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.701810 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="023f756f-dd53-42b9-a4ab-7411cb56ecef" containerName="registry" Aug 04 16:02:01.704765 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.704748 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.709024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709007 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Aug 04 16:02:01.709365 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709348 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Aug 04 16:02:01.709546 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709529 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Aug 04 16:02:01.709615 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709594 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Aug 04 16:02:01.709674 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709606 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Aug 04 16:02:01.709674 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709622 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Aug 04 16:02:01.709960 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709945 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Aug 04 16:02:01.710002 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.709971 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Aug 04 16:02:01.710505 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.710483 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nnrzr\"" Aug 04 16:02:01.716155 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.716137 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Aug 04 16:02:01.722855 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.722837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/006a0e30-830b-422d-926b-1a8a51842d8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.722936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.722869 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.722936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.722886 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.722936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.722906 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/006a0e30-830b-422d-926b-1a8a51842d8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723062 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.722962 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723062 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723014 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723062 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723051 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/006a0e30-830b-422d-926b-1a8a51842d8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723076 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723098 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723115 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/006a0e30-830b-422d-926b-1a8a51842d8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnn5\" (UniqueName: \"kubernetes.io/projected/006a0e30-830b-422d-926b-1a8a51842d8f-kube-api-access-mfnn5\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723322 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-web-config\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.723322 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.723225 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/006a0e30-830b-422d-926b-1a8a51842d8f-config-out\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.730994 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.730974 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:02:01.824512 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824473 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824512 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824512 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/006a0e30-830b-422d-926b-1a8a51842d8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnn5\" (UniqueName: \"kubernetes.io/projected/006a0e30-830b-422d-926b-1a8a51842d8f-kube-api-access-mfnn5\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824565 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-web-config\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/006a0e30-830b-422d-926b-1a8a51842d8f-config-out\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824650 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/006a0e30-830b-422d-926b-1a8a51842d8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824690 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824717 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.824756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/006a0e30-830b-422d-926b-1a8a51842d8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.825109 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.825109 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.825109 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/006a0e30-830b-422d-926b-1a8a51842d8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.825109 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.824923 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.825718 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.825674 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/006a0e30-830b-422d-926b-1a8a51842d8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.825998 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.825970 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/006a0e30-830b-422d-926b-1a8a51842d8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.826732 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.826706 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/006a0e30-830b-422d-926b-1a8a51842d8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.827745 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.827657 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/006a0e30-830b-422d-926b-1a8a51842d8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.827995 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.827943 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.828120 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.828089 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.828240 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.828190 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.828304 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.828239 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.828535 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.828519 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-web-config\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.828621 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.828599 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.829028 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.829012 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/006a0e30-830b-422d-926b-1a8a51842d8f-config-out\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.829765 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.829748 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/006a0e30-830b-422d-926b-1a8a51842d8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:01.841335 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:01.841306 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnn5\" (UniqueName: \"kubernetes.io/projected/006a0e30-830b-422d-926b-1a8a51842d8f-kube-api-access-mfnn5\") pod \"alertmanager-main-0\" (UID: \"006a0e30-830b-422d-926b-1a8a51842d8f\") " pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:02.015588 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:02.015489 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Aug 04 16:02:02.160250 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:02.160216 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Aug 04 16:02:02.163899 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:02:02.163872 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006a0e30_830b_422d_926b_1a8a51842d8f.slice/crio-3b9317c9717c29721a22a91e843a20d7f8fc8ba98ddf9e6dbecd7534b5308530 WatchSource:0}: Error finding container 3b9317c9717c29721a22a91e843a20d7f8fc8ba98ddf9e6dbecd7534b5308530: Status 404 returned error can't find the container with id 3b9317c9717c29721a22a91e843a20d7f8fc8ba98ddf9e6dbecd7534b5308530 Aug 04 16:02:02.627350 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:02.627317 2571 generic.go:356] "Generic (PLEG): container finished" podID="006a0e30-830b-422d-926b-1a8a51842d8f" containerID="2ab5848f6e490089dee9f70a64405e6bd91ceab3835515fd3a7846ed70b29aca" exitCode=0 Aug 04 16:02:02.627721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:02.627361 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerDied","Data":"2ab5848f6e490089dee9f70a64405e6bd91ceab3835515fd3a7846ed70b29aca"} Aug 04 16:02:02.627721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:02.627445 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerStarted","Data":"3b9317c9717c29721a22a91e843a20d7f8fc8ba98ddf9e6dbecd7534b5308530"} Aug 04 16:02:03.362981 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.362944 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c16ae6-eb59-4b7f-9190-b6f5f6573428" path="/var/lib/kubelet/pods/a8c16ae6-eb59-4b7f-9190-b6f5f6573428/volumes" Aug 04 16:02:03.635323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.635233 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerStarted","Data":"0f77cd465f39ab49d9b3c5abd995cae94fc238be96d860fea42df9073d49c639"} Aug 04 16:02:03.635323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.635315 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerStarted","Data":"f042e1dc3e72146145a3940d24dc62842c692520ebca2d616d1d5679da9c27e7"} Aug 04 16:02:03.635708 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.635353 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerStarted","Data":"bc15f0c3a50d130306bbab328690f4f89c3f04a47faf53dbe0c887ea9eae4193"} Aug 04 16:02:03.635708 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.635394 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerStarted","Data":"c012bb3263ac30fb6b515b99e338e757d0b3673dd42733d08f4cf88d8e425902"} Aug 04 16:02:03.635708 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.635433 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerStarted","Data":"9945bc24a12fda4cf357a30e69af84a996e86779f556ed677b55ae14a9c8a4eb"} Aug 04 16:02:03.635708 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.635471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"006a0e30-830b-422d-926b-1a8a51842d8f","Type":"ContainerStarted","Data":"20de3ea0d825920aaef0452befaa0e7fa6ce6c79b7d04d139f10b0560b8bd2d9"} Aug 04 16:02:03.668116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.668061 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.64835813 podStartE2EDuration="2.668042753s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.684623ms" isStatefulPod=false podCreationTimestamp="2026-08-04 16:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:02:03.666415563 +0000 UTC m=+332.760015811" watchObservedRunningTime="2026-08-04 16:02:03.668042753 +0000 UTC m=+332.761643000" Aug 04 16:02:03.831835 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:03.831794 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:07.133832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:07.133773 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:07.205826 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:07.205766 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:08.732171 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:08.732118 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:02:08.732665 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:08.732233 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:02:08.737046 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:08.737022 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:02:09.667273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:09.667244 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:02:09.858603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:09.858560 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:09.959275 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:09.959176 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:10.045301 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:10.045275 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:16.915536 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:16.915494 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:17.014662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:17.014624 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:02:17.300295 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:02:17.300196 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:44.428399 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:44.428366 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:44.516446 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:44.516417 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:45.109271 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.109225 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7"] Aug 04 16:04:45.111656 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.111633 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.114998 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.114975 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nt86t\"" Aug 04 16:04:45.115134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.115016 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 16:04:45.115134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.115021 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 16:04:45.134863 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.134831 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7"] Aug 04 16:04:45.275864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.275812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.275864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.275865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.276095 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.275928 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5d6v\" (UniqueName: \"kubernetes.io/projected/152fbba5-67ad-4022-bdba-de8950996a95-kube-api-access-r5d6v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.377024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.376929 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.377024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.376965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.377024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.376994 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5d6v\" (UniqueName: \"kubernetes.io/projected/152fbba5-67ad-4022-bdba-de8950996a95-kube-api-access-r5d6v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.377442 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.377416 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.377483 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.377421 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.385972 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.385945 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5d6v\" (UniqueName: \"kubernetes.io/projected/152fbba5-67ad-4022-bdba-de8950996a95-kube-api-access-r5d6v\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.421948 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.421914 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:04:45.556642 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.556612 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7"] Aug 04 16:04:45.559143 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:04:45.559113 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152fbba5_67ad_4022_bdba_de8950996a95.slice/crio-9456176976ca35eda71801ed80472160931d72d8acbe46d8222339479dc40f05 WatchSource:0}: Error finding container 9456176976ca35eda71801ed80472160931d72d8acbe46d8222339479dc40f05: Status 404 returned error can't find the container with id 9456176976ca35eda71801ed80472160931d72d8acbe46d8222339479dc40f05 Aug 04 16:04:45.561435 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:45.561414 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:04:46.294726 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:46.294686 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" event={"ID":"152fbba5-67ad-4022-bdba-de8950996a95","Type":"ContainerStarted","Data":"9456176976ca35eda71801ed80472160931d72d8acbe46d8222339479dc40f05"} Aug 04 16:04:47.585944 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:47.585905 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:47.686136 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:47.686087 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:52.327159 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:52.327121 2571 generic.go:356] "Generic (PLEG): container finished" podID="152fbba5-67ad-4022-bdba-de8950996a95" containerID="a6ba8928b5e767a80c86b954417d65c21db2c563493456d8000c123be802a737" exitCode=0 Aug 04 16:04:52.327572 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:52.327222 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" event={"ID":"152fbba5-67ad-4022-bdba-de8950996a95","Type":"ContainerDied","Data":"a6ba8928b5e767a80c86b954417d65c21db2c563493456d8000c123be802a737"} Aug 04 16:04:55.343105 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:55.343065 2571 generic.go:356] "Generic (PLEG): container finished" podID="152fbba5-67ad-4022-bdba-de8950996a95" containerID="248bb4fd928eed3a474b5438915bb2b386d0a0ee6dfc905e12b477d87ced3ffb" exitCode=0 Aug 04 16:04:55.343529 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:04:55.343156 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" event={"ID":"152fbba5-67ad-4022-bdba-de8950996a95","Type":"ContainerDied","Data":"248bb4fd928eed3a474b5438915bb2b386d0a0ee6dfc905e12b477d87ced3ffb"} Aug 04 16:05:02.377307 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:02.377270 2571 generic.go:356] "Generic (PLEG): container finished" podID="152fbba5-67ad-4022-bdba-de8950996a95" containerID="1cb4ea1bc15bd1db548b104eb338e894497615ca8b01f883dce76d98b4693748" exitCode=0 Aug 04 16:05:02.377691 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:02.377340 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" event={"ID":"152fbba5-67ad-4022-bdba-de8950996a95","Type":"ContainerDied","Data":"1cb4ea1bc15bd1db548b104eb338e894497615ca8b01f883dce76d98b4693748"} Aug 04 16:05:03.509710 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.509682 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:05:03.547611 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.547573 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5d6v\" (UniqueName: \"kubernetes.io/projected/152fbba5-67ad-4022-bdba-de8950996a95-kube-api-access-r5d6v\") pod \"152fbba5-67ad-4022-bdba-de8950996a95\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " Aug 04 16:05:03.547808 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.547676 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-bundle\") pod \"152fbba5-67ad-4022-bdba-de8950996a95\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " Aug 04 16:05:03.547808 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.547706 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-util\") pod \"152fbba5-67ad-4022-bdba-de8950996a95\" (UID: \"152fbba5-67ad-4022-bdba-de8950996a95\") " Aug 04 16:05:03.548597 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.548570 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-bundle" (OuterVolumeSpecName: "bundle") pod "152fbba5-67ad-4022-bdba-de8950996a95" (UID: "152fbba5-67ad-4022-bdba-de8950996a95"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:05:03.549951 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.549927 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152fbba5-67ad-4022-bdba-de8950996a95-kube-api-access-r5d6v" (OuterVolumeSpecName: "kube-api-access-r5d6v") pod "152fbba5-67ad-4022-bdba-de8950996a95" (UID: "152fbba5-67ad-4022-bdba-de8950996a95"). InnerVolumeSpecName "kube-api-access-r5d6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:05:03.553806 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.553768 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-util" (OuterVolumeSpecName: "util") pod "152fbba5-67ad-4022-bdba-de8950996a95" (UID: "152fbba5-67ad-4022-bdba-de8950996a95"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:05:03.649320 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.649230 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:03.649320 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.649263 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/152fbba5-67ad-4022-bdba-de8950996a95-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:03.649320 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:03.649273 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5d6v\" (UniqueName: \"kubernetes.io/projected/152fbba5-67ad-4022-bdba-de8950996a95-kube-api-access-r5d6v\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:04.388986 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:04.388956 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" Aug 04 16:05:04.388986 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:04.388976 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540xqnq7" event={"ID":"152fbba5-67ad-4022-bdba-de8950996a95","Type":"ContainerDied","Data":"9456176976ca35eda71801ed80472160931d72d8acbe46d8222339479dc40f05"} Aug 04 16:05:04.389191 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:04.389023 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9456176976ca35eda71801ed80472160931d72d8acbe46d8222339479dc40f05" Aug 04 16:05:14.155731 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:14.155697 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:14.256601 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:14.256563 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:17.189820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.189780 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59"] Aug 04 16:05:17.190272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.190163 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152fbba5-67ad-4022-bdba-de8950996a95" containerName="extract" Aug 04 16:05:17.190272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.190176 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="152fbba5-67ad-4022-bdba-de8950996a95" containerName="extract" Aug 04 16:05:17.190272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.190186 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152fbba5-67ad-4022-bdba-de8950996a95" containerName="pull" Aug 04 16:05:17.190272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.190195 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="152fbba5-67ad-4022-bdba-de8950996a95" containerName="pull" Aug 04 16:05:17.190272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.190227 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152fbba5-67ad-4022-bdba-de8950996a95" containerName="util" Aug 04 16:05:17.190272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.190234 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="152fbba5-67ad-4022-bdba-de8950996a95" containerName="util" Aug 04 16:05:17.190469 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.190294 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="152fbba5-67ad-4022-bdba-de8950996a95" containerName="extract" Aug 04 16:05:17.194424 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.194401 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.197418 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.197393 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Aug 04 16:05:17.197541 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.197394 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:05:17.197828 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.197813 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hzts4\"" Aug 04 16:05:17.210661 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.210632 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59"] Aug 04 16:05:17.270903 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.270858 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-69t59\" (UID: \"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.271081 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.270929 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8f6x\" (UniqueName: \"kubernetes.io/projected/a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5-kube-api-access-k8f6x\") pod \"cert-manager-operator-controller-manager-55c444bf86-69t59\" (UID: \"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.371575 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.371535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-69t59\" (UID: \"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.371713 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.371610 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8f6x\" (UniqueName: \"kubernetes.io/projected/a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5-kube-api-access-k8f6x\") pod \"cert-manager-operator-controller-manager-55c444bf86-69t59\" (UID: \"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.371959 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.371935 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-69t59\" (UID: \"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.381178 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.381139 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8f6x\" (UniqueName: \"kubernetes.io/projected/a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5-kube-api-access-k8f6x\") pod \"cert-manager-operator-controller-manager-55c444bf86-69t59\" (UID: \"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.504055 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.503951 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" Aug 04 16:05:17.648749 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.648720 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59"] Aug 04 16:05:17.651723 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:05:17.651694 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e2e36c_49cf_4b89_8e13_e6fb78e8e1b5.slice/crio-bfa65c2a401bb44bf5d2402c03fbf883d44b38a26194987b96c40896ab3c6850 WatchSource:0}: Error finding container bfa65c2a401bb44bf5d2402c03fbf883d44b38a26194987b96c40896ab3c6850: Status 404 returned error can't find the container with id bfa65c2a401bb44bf5d2402c03fbf883d44b38a26194987b96c40896ab3c6850 Aug 04 16:05:17.957612 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:17.957575 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:18.447826 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:18.447789 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" event={"ID":"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5","Type":"ContainerStarted","Data":"bfa65c2a401bb44bf5d2402c03fbf883d44b38a26194987b96c40896ab3c6850"} Aug 04 16:05:18.775421 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:18.775328 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:18.879495 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:18.879460 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:19.455971 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:19.455937 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" event={"ID":"a5e2e36c-49cf-4b89-8e13-e6fb78e8e1b5","Type":"ContainerStarted","Data":"a86eacaf01fe4a3f52684f31508a60b36a765c3eb1ed18b992956ee47a46c972"} Aug 04 16:05:19.949078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:19.949028 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:20.088528 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.088487 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:20.378531 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.378485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:20.530696 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.530666 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:20.888895 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.888836 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-69t59" podStartSLOduration=2.195841502 podStartE2EDuration="3.888817612s" totalImagesPullingTime="1.69297611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:05:19.49311136 +0000 UTC m=+528.586711610" watchObservedRunningTime="2026-08-04 16:05:20.888817612 +0000 UTC m=+529.982417875" Aug 04 16:05:20.889198 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.889182 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh"] Aug 04 16:05:20.891836 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.891818 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:20.894912 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.894889 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nt86t\"" Aug 04 16:05:20.895297 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.895282 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 16:05:20.895981 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.895963 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 16:05:20.906526 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:20.906493 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh"] Aug 04 16:05:21.005120 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.005074 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.005120 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.005124 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8bc\" (UniqueName: \"kubernetes.io/projected/b594f6e4-e844-4184-adea-d43c96ec6abb-kube-api-access-gl8bc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.005366 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.005149 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.106045 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.106003 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.106045 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.106050 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8bc\" (UniqueName: \"kubernetes.io/projected/b594f6e4-e844-4184-adea-d43c96ec6abb-kube-api-access-gl8bc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.106281 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.106070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.106472 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.106451 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.106509 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.106479 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.116303 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.116278 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8bc\" (UniqueName: \"kubernetes.io/projected/b594f6e4-e844-4184-adea-d43c96ec6abb-kube-api-access-gl8bc\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.201172 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.201074 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:21.341580 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.341556 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh"] Aug 04 16:05:21.343933 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:05:21.343899 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb594f6e4_e844_4184_adea_d43c96ec6abb.slice/crio-0c585f61a4269f1bc682724c312304e427ba9bfc909de6a70190a7d7cf0d2c44 WatchSource:0}: Error finding container 0c585f61a4269f1bc682724c312304e427ba9bfc909de6a70190a7d7cf0d2c44: Status 404 returned error can't find the container with id 0c585f61a4269f1bc682724c312304e427ba9bfc909de6a70190a7d7cf0d2c44 Aug 04 16:05:21.465611 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.465508 2571 generic.go:356] "Generic (PLEG): container finished" podID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerID="16599e5010fa5f40203e2f6d10d464b90e8ef3f79ecc77b1730ba5e3bf73e2d4" exitCode=0 Aug 04 16:05:21.465611 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.465587 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" event={"ID":"b594f6e4-e844-4184-adea-d43c96ec6abb","Type":"ContainerDied","Data":"16599e5010fa5f40203e2f6d10d464b90e8ef3f79ecc77b1730ba5e3bf73e2d4"} Aug 04 16:05:21.465802 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:21.465650 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" event={"ID":"b594f6e4-e844-4184-adea-d43c96ec6abb","Type":"ContainerStarted","Data":"0c585f61a4269f1bc682724c312304e427ba9bfc909de6a70190a7d7cf0d2c44"} Aug 04 16:05:22.776814 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:22.776768 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:23.190684 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.190650 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x"] Aug 04 16:05:23.192862 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.192845 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.198087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.198057 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Aug 04 16:05:23.199362 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.199288 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-mgf8b\"" Aug 04 16:05:23.199362 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.199288 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Aug 04 16:05:23.208793 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.208760 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x"] Aug 04 16:05:23.298567 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.298504 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:23.327734 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.327695 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab21e16f-7dbb-4991-b84d-7d278889bca1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9wj6x\" (UID: \"ab21e16f-7dbb-4991-b84d-7d278889bca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.327897 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.327756 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5fc\" (UniqueName: \"kubernetes.io/projected/ab21e16f-7dbb-4991-b84d-7d278889bca1-kube-api-access-fq5fc\") pod \"cert-manager-cainjector-559cdd8c87-9wj6x\" (UID: \"ab21e16f-7dbb-4991-b84d-7d278889bca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.428979 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.428942 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab21e16f-7dbb-4991-b84d-7d278889bca1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9wj6x\" (UID: \"ab21e16f-7dbb-4991-b84d-7d278889bca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.429196 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.429021 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5fc\" (UniqueName: \"kubernetes.io/projected/ab21e16f-7dbb-4991-b84d-7d278889bca1-kube-api-access-fq5fc\") pod \"cert-manager-cainjector-559cdd8c87-9wj6x\" (UID: \"ab21e16f-7dbb-4991-b84d-7d278889bca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.448836 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.448753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5fc\" (UniqueName: \"kubernetes.io/projected/ab21e16f-7dbb-4991-b84d-7d278889bca1-kube-api-access-fq5fc\") pod \"cert-manager-cainjector-559cdd8c87-9wj6x\" (UID: \"ab21e16f-7dbb-4991-b84d-7d278889bca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.449858 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.449836 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ab21e16f-7dbb-4991-b84d-7d278889bca1-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9wj6x\" (UID: \"ab21e16f-7dbb-4991-b84d-7d278889bca1\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.515733 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.515691 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" Aug 04 16:05:23.653563 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:23.653524 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x"] Aug 04 16:05:23.656722 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:05:23.656691 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab21e16f_7dbb_4991_b84d_7d278889bca1.slice/crio-4b60ad47f574a4446c00333be3f4221ee550ef6301653c5c1e5891f33af7964d WatchSource:0}: Error finding container 4b60ad47f574a4446c00333be3f4221ee550ef6301653c5c1e5891f33af7964d: Status 404 returned error can't find the container with id 4b60ad47f574a4446c00333be3f4221ee550ef6301653c5c1e5891f33af7964d Aug 04 16:05:24.482297 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:24.482249 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" event={"ID":"ab21e16f-7dbb-4991-b84d-7d278889bca1","Type":"ContainerStarted","Data":"4b60ad47f574a4446c00333be3f4221ee550ef6301653c5c1e5891f33af7964d"} Aug 04 16:05:26.145385 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:26.145351 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:26.217771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:26.217731 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:26.496517 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:26.496421 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" event={"ID":"b594f6e4-e844-4184-adea-d43c96ec6abb","Type":"ContainerStarted","Data":"ef7ddf738b03616da2bf40c0f4a3a334fb74fa1aa54c4023e2f08295f18a4da8"} Aug 04 16:05:27.502854 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:27.502810 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" event={"ID":"ab21e16f-7dbb-4991-b84d-7d278889bca1","Type":"ContainerStarted","Data":"58121dd65833550ec1fb02f467c0686ba1f6bb193b52c1996ecfa1cc36135282"} Aug 04 16:05:27.504701 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:27.504672 2571 generic.go:356] "Generic (PLEG): container finished" podID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerID="ef7ddf738b03616da2bf40c0f4a3a334fb74fa1aa54c4023e2f08295f18a4da8" exitCode=0 Aug 04 16:05:27.504812 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:27.504717 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" event={"ID":"b594f6e4-e844-4184-adea-d43c96ec6abb","Type":"ContainerDied","Data":"ef7ddf738b03616da2bf40c0f4a3a334fb74fa1aa54c4023e2f08295f18a4da8"} Aug 04 16:05:27.540718 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:27.540663 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9wj6x" podStartSLOduration=1.5298421960000002 podStartE2EDuration="4.540647531s" totalImagesPullingTime="3.010805335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:05:27.538414009 +0000 UTC m=+536.632014257" watchObservedRunningTime="2026-08-04 16:05:27.540647531 +0000 UTC m=+536.634247779" Aug 04 16:05:28.511323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:28.511284 2571 generic.go:356] "Generic (PLEG): container finished" podID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerID="b043c3ea75b35dcc3b7fdc553c3df35052980f80cb46386893df577be8f74ee4" exitCode=0 Aug 04 16:05:28.511725 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:28.511374 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" event={"ID":"b594f6e4-e844-4184-adea-d43c96ec6abb","Type":"ContainerDied","Data":"b043c3ea75b35dcc3b7fdc553c3df35052980f80cb46386893df577be8f74ee4"} Aug 04 16:05:29.640364 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.640338 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:29.786830 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.786734 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-util\") pod \"b594f6e4-e844-4184-adea-d43c96ec6abb\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " Aug 04 16:05:29.786830 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.786813 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-bundle\") pod \"b594f6e4-e844-4184-adea-d43c96ec6abb\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " Aug 04 16:05:29.787073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.786854 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl8bc\" (UniqueName: \"kubernetes.io/projected/b594f6e4-e844-4184-adea-d43c96ec6abb-kube-api-access-gl8bc\") pod \"b594f6e4-e844-4184-adea-d43c96ec6abb\" (UID: \"b594f6e4-e844-4184-adea-d43c96ec6abb\") " Aug 04 16:05:29.787317 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.787289 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-bundle" (OuterVolumeSpecName: "bundle") pod "b594f6e4-e844-4184-adea-d43c96ec6abb" (UID: "b594f6e4-e844-4184-adea-d43c96ec6abb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:05:29.789049 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.789027 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b594f6e4-e844-4184-adea-d43c96ec6abb-kube-api-access-gl8bc" (OuterVolumeSpecName: "kube-api-access-gl8bc") pod "b594f6e4-e844-4184-adea-d43c96ec6abb" (UID: "b594f6e4-e844-4184-adea-d43c96ec6abb"). InnerVolumeSpecName "kube-api-access-gl8bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:05:29.791490 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.791445 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-util" (OuterVolumeSpecName: "util") pod "b594f6e4-e844-4184-adea-d43c96ec6abb" (UID: "b594f6e4-e844-4184-adea-d43c96ec6abb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:05:29.888253 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.888214 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:29.888253 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.888247 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b594f6e4-e844-4184-adea-d43c96ec6abb-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:29.888253 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:29.888257 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl8bc\" (UniqueName: \"kubernetes.io/projected/b594f6e4-e844-4184-adea-d43c96ec6abb-kube-api-access-gl8bc\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:30.405497 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:30.405454 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:30.522134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:30.522096 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" event={"ID":"b594f6e4-e844-4184-adea-d43c96ec6abb","Type":"ContainerDied","Data":"0c585f61a4269f1bc682724c312304e427ba9bfc909de6a70190a7d7cf0d2c44"} Aug 04 16:05:30.522326 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:30.522144 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f64dnh" Aug 04 16:05:30.522326 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:30.522154 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c585f61a4269f1bc682724c312304e427ba9bfc909de6a70190a7d7cf0d2c44" Aug 04 16:05:33.863860 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:33.863816 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:33.969065 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:33.969026 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:35.424849 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.424814 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch"] Aug 04 16:05:35.425353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.425125 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerName="pull" Aug 04 16:05:35.425353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.425136 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerName="pull" Aug 04 16:05:35.425353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.425152 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerName="util" Aug 04 16:05:35.425353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.425158 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerName="util" Aug 04 16:05:35.425353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.425164 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerName="extract" Aug 04 16:05:35.425353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.425169 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerName="extract" Aug 04 16:05:35.425776 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.425736 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b594f6e4-e844-4184-adea-d43c96ec6abb" containerName="extract" Aug 04 16:05:35.432332 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.432293 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.436660 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.436637 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-9fmpw\"" Aug 04 16:05:35.436803 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.436640 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Aug 04 16:05:35.436803 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.436640 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Aug 04 16:05:35.446578 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.446552 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch"] Aug 04 16:05:35.532863 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.532820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a67e72cf-b697-40b3-a754-11e251b5999e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nx2ch\" (UID: \"a67e72cf-b697-40b3-a754-11e251b5999e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.533058 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.532899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbptd\" (UniqueName: \"kubernetes.io/projected/a67e72cf-b697-40b3-a754-11e251b5999e-kube-api-access-tbptd\") pod \"openshift-lws-operator-fd8ccff4c-nx2ch\" (UID: \"a67e72cf-b697-40b3-a754-11e251b5999e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.634069 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.634033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a67e72cf-b697-40b3-a754-11e251b5999e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nx2ch\" (UID: \"a67e72cf-b697-40b3-a754-11e251b5999e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.634289 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.634197 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbptd\" (UniqueName: \"kubernetes.io/projected/a67e72cf-b697-40b3-a754-11e251b5999e-kube-api-access-tbptd\") pod \"openshift-lws-operator-fd8ccff4c-nx2ch\" (UID: \"a67e72cf-b697-40b3-a754-11e251b5999e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.634451 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.634433 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a67e72cf-b697-40b3-a754-11e251b5999e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nx2ch\" (UID: \"a67e72cf-b697-40b3-a754-11e251b5999e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.644010 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.643975 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbptd\" (UniqueName: \"kubernetes.io/projected/a67e72cf-b697-40b3-a754-11e251b5999e-kube-api-access-tbptd\") pod \"openshift-lws-operator-fd8ccff4c-nx2ch\" (UID: \"a67e72cf-b697-40b3-a754-11e251b5999e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.742488 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.742396 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" Aug 04 16:05:35.888788 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:35.888628 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch"] Aug 04 16:05:35.891155 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:05:35.891125 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda67e72cf_b697_40b3_a754_11e251b5999e.slice/crio-b54916a2c76742ffd508f0d8867bf1ce245b41964b03a023362d034f9e783885 WatchSource:0}: Error finding container b54916a2c76742ffd508f0d8867bf1ce245b41964b03a023362d034f9e783885: Status 404 returned error can't find the container with id b54916a2c76742ffd508f0d8867bf1ce245b41964b03a023362d034f9e783885 Aug 04 16:05:36.551011 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:36.550971 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" event={"ID":"a67e72cf-b697-40b3-a754-11e251b5999e","Type":"ContainerStarted","Data":"b54916a2c76742ffd508f0d8867bf1ce245b41964b03a023362d034f9e783885"} Aug 04 16:05:38.564344 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:38.564303 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" event={"ID":"a67e72cf-b697-40b3-a754-11e251b5999e","Type":"ContainerStarted","Data":"4030703f09b945e9849de8c3b4bd9b6f0473dcd7ef43259637c16913afbcbb2b"} Aug 04 16:05:38.584673 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:38.584614 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nx2ch" podStartSLOduration=1.825131314 podStartE2EDuration="3.584590691s" totalImagesPullingTime="1.759459377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:05:38.584279653 +0000 UTC m=+547.677879909" watchObservedRunningTime="2026-08-04 16:05:38.584590691 +0000 UTC m=+547.678190943" Aug 04 16:05:39.354694 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:39.354654 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:39.467747 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:39.467714 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:41.120079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.120037 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf"] Aug 04 16:05:41.123977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.123951 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.127440 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.127415 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nt86t\"" Aug 04 16:05:41.127570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.127420 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 16:05:41.128432 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.128417 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 16:05:41.134937 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.134901 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf"] Aug 04 16:05:41.183531 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.183489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.183531 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.183537 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrtd\" (UniqueName: \"kubernetes.io/projected/f0bcb4cc-8619-4903-831f-18116317d475-kube-api-access-rtrtd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.183759 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.183638 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.284132 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.284090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.284344 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.284146 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.284344 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.284182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrtd\" (UniqueName: \"kubernetes.io/projected/f0bcb4cc-8619-4903-831f-18116317d475-kube-api-access-rtrtd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.284526 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.284505 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.284603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.284581 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.294867 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.294842 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrtd\" (UniqueName: \"kubernetes.io/projected/f0bcb4cc-8619-4903-831f-18116317d475-kube-api-access-rtrtd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.435248 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.435127 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:41.586159 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:41.586134 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf"] Aug 04 16:05:41.588652 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:05:41.588627 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bcb4cc_8619_4903_831f_18116317d475.slice/crio-d0b6ee8dcdc35d056a478bd3df5d5a1074ad476fb65097dd3ff8817d562cd14c WatchSource:0}: Error finding container d0b6ee8dcdc35d056a478bd3df5d5a1074ad476fb65097dd3ff8817d562cd14c: Status 404 returned error can't find the container with id d0b6ee8dcdc35d056a478bd3df5d5a1074ad476fb65097dd3ff8817d562cd14c Aug 04 16:05:42.584259 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:42.584223 2571 generic.go:356] "Generic (PLEG): container finished" podID="f0bcb4cc-8619-4903-831f-18116317d475" containerID="012a6a1cc2e3b6857e06114b3009fdb5d6058d845b573e1c57295891d2618759" exitCode=0 Aug 04 16:05:42.584693 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:42.584283 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" event={"ID":"f0bcb4cc-8619-4903-831f-18116317d475","Type":"ContainerDied","Data":"012a6a1cc2e3b6857e06114b3009fdb5d6058d845b573e1c57295891d2618759"} Aug 04 16:05:42.584693 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:42.584338 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" event={"ID":"f0bcb4cc-8619-4903-831f-18116317d475","Type":"ContainerStarted","Data":"d0b6ee8dcdc35d056a478bd3df5d5a1074ad476fb65097dd3ff8817d562cd14c"} Aug 04 16:05:42.941704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:42.941620 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:43.591328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:43.591285 2571 generic.go:356] "Generic (PLEG): container finished" podID="f0bcb4cc-8619-4903-831f-18116317d475" containerID="d46a5a39c077c99ae15720eae2950befe6285ace7f2c67b9b37ad51f19abcedb" exitCode=0 Aug 04 16:05:43.591727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:43.591355 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" event={"ID":"f0bcb4cc-8619-4903-831f-18116317d475","Type":"ContainerDied","Data":"d46a5a39c077c99ae15720eae2950befe6285ace7f2c67b9b37ad51f19abcedb"} Aug 04 16:05:44.197323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:44.197291 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:44.295278 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:44.295240 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:44.598339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:44.598299 2571 generic.go:356] "Generic (PLEG): container finished" podID="f0bcb4cc-8619-4903-831f-18116317d475" containerID="e59a961a354db28874b913cf4dccd8d4334a82491c44672f8b161952c291abfb" exitCode=0 Aug 04 16:05:44.598810 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:44.598387 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" event={"ID":"f0bcb4cc-8619-4903-831f-18116317d475","Type":"ContainerDied","Data":"e59a961a354db28874b913cf4dccd8d4334a82491c44672f8b161952c291abfb"} Aug 04 16:05:45.732716 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.732687 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:45.829450 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.829411 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-bundle\") pod \"f0bcb4cc-8619-4903-831f-18116317d475\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " Aug 04 16:05:45.829686 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.829458 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtrtd\" (UniqueName: \"kubernetes.io/projected/f0bcb4cc-8619-4903-831f-18116317d475-kube-api-access-rtrtd\") pod \"f0bcb4cc-8619-4903-831f-18116317d475\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " Aug 04 16:05:45.829686 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.829485 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-util\") pod \"f0bcb4cc-8619-4903-831f-18116317d475\" (UID: \"f0bcb4cc-8619-4903-831f-18116317d475\") " Aug 04 16:05:45.830290 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.830260 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-bundle" (OuterVolumeSpecName: "bundle") pod "f0bcb4cc-8619-4903-831f-18116317d475" (UID: "f0bcb4cc-8619-4903-831f-18116317d475"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:05:45.831646 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.831621 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bcb4cc-8619-4903-831f-18116317d475-kube-api-access-rtrtd" (OuterVolumeSpecName: "kube-api-access-rtrtd") pod "f0bcb4cc-8619-4903-831f-18116317d475" (UID: "f0bcb4cc-8619-4903-831f-18116317d475"). InnerVolumeSpecName "kube-api-access-rtrtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:05:45.837405 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.837348 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-util" (OuterVolumeSpecName: "util") pod "f0bcb4cc-8619-4903-831f-18116317d475" (UID: "f0bcb4cc-8619-4903-831f-18116317d475"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:05:45.930918 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.930817 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:45.930918 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.930856 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtrtd\" (UniqueName: \"kubernetes.io/projected/f0bcb4cc-8619-4903-831f-18116317d475-kube-api-access-rtrtd\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:45.930918 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:45.930868 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0bcb4cc-8619-4903-831f-18116317d475-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:05:46.610958 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:46.610926 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" Aug 04 16:05:46.610958 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:46.610941 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvqwrf" event={"ID":"f0bcb4cc-8619-4903-831f-18116317d475","Type":"ContainerDied","Data":"d0b6ee8dcdc35d056a478bd3df5d5a1074ad476fb65097dd3ff8817d562cd14c"} Aug 04 16:05:46.611230 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:46.610987 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0b6ee8dcdc35d056a478bd3df5d5a1074ad476fb65097dd3ff8817d562cd14c" Aug 04 16:05:57.386557 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:57.386522 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:57.526231 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:57.526162 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:57.833439 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:57.833401 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:58.216179 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.216097 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:58.359164 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.359127 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:58.418765 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.418730 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q"] Aug 04 16:05:58.421924 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.419616 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bcb4cc-8619-4903-831f-18116317d475" containerName="util" Aug 04 16:05:58.421924 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.419655 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bcb4cc-8619-4903-831f-18116317d475" containerName="util" Aug 04 16:05:58.421924 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.419670 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bcb4cc-8619-4903-831f-18116317d475" containerName="extract" Aug 04 16:05:58.421924 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.419681 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bcb4cc-8619-4903-831f-18116317d475" containerName="extract" Aug 04 16:05:58.421924 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.419728 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bcb4cc-8619-4903-831f-18116317d475" containerName="pull" Aug 04 16:05:58.421924 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.419744 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bcb4cc-8619-4903-831f-18116317d475" containerName="pull" Aug 04 16:05:58.421924 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.419940 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bcb4cc-8619-4903-831f-18116317d475" containerName="extract" Aug 04 16:05:58.424592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.424504 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.493272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.493159 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 16:05:58.493464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.493415 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 16:05:58.493464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.493436 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ft4sg\"" Aug 04 16:05:58.493587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.493536 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 16:05:58.493587 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.493536 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 16:05:58.495750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.495722 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q"] Aug 04 16:05:58.540289 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.540239 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbd0d08e-cbbd-4937-9858-a698d11eed1e-manifests\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.540475 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.540330 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbd0d08e-cbbd-4937-9858-a698d11eed1e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.540475 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.540371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd0d08e-cbbd-4937-9858-a698d11eed1e-tmp\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.540475 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.540388 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xtwf\" (UniqueName: \"kubernetes.io/projected/bbd0d08e-cbbd-4937-9858-a698d11eed1e-kube-api-access-8xtwf\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.540475 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.540417 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbd0d08e-cbbd-4937-9858-a698d11eed1e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.641518 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.641467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbd0d08e-cbbd-4937-9858-a698d11eed1e-manifests\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.641709 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.641536 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbd0d08e-cbbd-4937-9858-a698d11eed1e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.641709 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.641591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd0d08e-cbbd-4937-9858-a698d11eed1e-tmp\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.641709 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.641612 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xtwf\" (UniqueName: \"kubernetes.io/projected/bbd0d08e-cbbd-4937-9858-a698d11eed1e-kube-api-access-8xtwf\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.641709 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.641635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbd0d08e-cbbd-4937-9858-a698d11eed1e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.641974 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.641946 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bbd0d08e-cbbd-4937-9858-a698d11eed1e-manifests\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.644058 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.644021 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbd0d08e-cbbd-4937-9858-a698d11eed1e-tmp\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.644398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.644376 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbd0d08e-cbbd-4937-9858-a698d11eed1e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.644508 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.644380 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bbd0d08e-cbbd-4937-9858-a698d11eed1e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.659502 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.659461 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xtwf\" (UniqueName: \"kubernetes.io/projected/bbd0d08e-cbbd-4937-9858-a698d11eed1e-kube-api-access-8xtwf\") pod \"opendatahub-operator-controller-manager-7dfb598967-sps7q\" (UID: \"bbd0d08e-cbbd-4937-9858-a698d11eed1e\") " pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.735635 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.735589 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:05:58.892596 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.892559 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q"] Aug 04 16:05:58.895799 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:05:58.895765 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd0d08e_cbbd_4937_9858_a698d11eed1e.slice/crio-262187d752b0f8d017e40abe85e8bb09afba55c0ac4623014ec02954f7c3867f WatchSource:0}: Error finding container 262187d752b0f8d017e40abe85e8bb09afba55c0ac4623014ec02954f7c3867f: Status 404 returned error can't find the container with id 262187d752b0f8d017e40abe85e8bb09afba55c0ac4623014ec02954f7c3867f Aug 04 16:05:58.996791 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:58.996756 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:59.104918 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:59.104875 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:59.194296 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:59.194262 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:59.677258 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:59.677190 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" event={"ID":"bbd0d08e-cbbd-4937-9858-a698d11eed1e","Type":"ContainerStarted","Data":"262187d752b0f8d017e40abe85e8bb09afba55c0ac4623014ec02954f7c3867f"} Aug 04 16:05:59.860836 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:59.860802 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:59.977329 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:05:59.977246 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:04.710268 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:04.710230 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" event={"ID":"bbd0d08e-cbbd-4937-9858-a698d11eed1e","Type":"ContainerStarted","Data":"d646e431eede506278668b0742ccc562b23b81640d54f20aa054372c21accb1c"} Aug 04 16:06:05.718890 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:05.718844 2571 generic.go:356] "Generic (PLEG): container finished" podID="bbd0d08e-cbbd-4937-9858-a698d11eed1e" containerID="d646e431eede506278668b0742ccc562b23b81640d54f20aa054372c21accb1c" exitCode=0 Aug 04 16:06:05.719345 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:05.718919 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" event={"ID":"bbd0d08e-cbbd-4937-9858-a698d11eed1e","Type":"ContainerDied","Data":"d646e431eede506278668b0742ccc562b23b81640d54f20aa054372c21accb1c"} Aug 04 16:06:06.726454 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:06.726415 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" event={"ID":"bbd0d08e-cbbd-4937-9858-a698d11eed1e","Type":"ContainerStarted","Data":"1928d0fbf262dd4be90f26d67be2c002f55998aec1b5ff093d29e4b88b0676e2"} Aug 04 16:06:06.726856 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:06.726562 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:06:06.765192 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:06.765083 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" podStartSLOduration=2.48086456 podStartE2EDuration="8.765065714s" totalImagesPullingTime="5.277742008s" totalInitContainerRuntime="1.006459146s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:05:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:06.763307201 +0000 UTC m=+575.856907450" watchObservedRunningTime="2026-08-04 16:06:06.765065714 +0000 UTC m=+575.858665962" Aug 04 16:06:13.481490 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:13.481422 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:13.612647 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:13.612608 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:17.733641 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:17.733610 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dfb598967-sps7q" Aug 04 16:06:19.576145 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:19.576108 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:19.756185 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:19.756148 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:20.233468 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.233422 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:20.418268 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.418227 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj"] Aug 04 16:06:20.426679 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.426652 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.435691 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.435663 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afa294db-fb48-47cd-84a8-2260c93aca2d-tmp\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.435836 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.435710 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xpt\" (UniqueName: \"kubernetes.io/projected/afa294db-fb48-47cd-84a8-2260c93aca2d-kube-api-access-j4xpt\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.435836 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.435782 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa294db-fb48-47cd-84a8-2260c93aca2d-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.441809 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.441786 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 16:06:20.441953 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.441856 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 16:06:20.441953 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.441886 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 16:06:20.444915 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.444881 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj"] Aug 04 16:06:20.447804 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.447775 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 16:06:20.452291 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.452272 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5zk85\"" Aug 04 16:06:20.502537 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.502455 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v6xnk"] Aug 04 16:06:20.505915 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.505893 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:20.510489 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.510472 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 16:06:20.511061 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.511037 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rk6dt\"" Aug 04 16:06:20.536885 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.536853 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afa294db-fb48-47cd-84a8-2260c93aca2d-tmp\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.537021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.536895 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xpt\" (UniqueName: \"kubernetes.io/projected/afa294db-fb48-47cd-84a8-2260c93aca2d-kube-api-access-j4xpt\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.537021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.536916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa294db-fb48-47cd-84a8-2260c93aca2d-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.537021 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:20.536997 2571 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Aug 04 16:06:20.537130 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.537039 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzr5\" (UniqueName: \"kubernetes.io/projected/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-kube-api-access-ztzr5\") pod \"odh-model-controller-7998c67dc4-v6xnk\" (UID: \"c2b7dc31-4af2-4db3-afd3-4fd6b644084f\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:20.537130 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:20.537077 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afa294db-fb48-47cd-84a8-2260c93aca2d-tls-certs podName:afa294db-fb48-47cd-84a8-2260c93aca2d nodeName:}" failed. No retries permitted until 2026-08-04 16:06:21.037058486 +0000 UTC m=+590.130658714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/afa294db-fb48-47cd-84a8-2260c93aca2d-tls-certs") pod "kube-auth-proxy-78b5c5f4c4-kx7qj" (UID: "afa294db-fb48-47cd-84a8-2260c93aca2d") : secret "kube-auth-proxy-tls" not found Aug 04 16:06:20.537252 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.537150 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-cert\") pod \"odh-model-controller-7998c67dc4-v6xnk\" (UID: \"c2b7dc31-4af2-4db3-afd3-4fd6b644084f\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:20.539117 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.539089 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/afa294db-fb48-47cd-84a8-2260c93aca2d-tmp\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.549160 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.549129 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v6xnk"] Aug 04 16:06:20.565229 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.563034 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xpt\" (UniqueName: \"kubernetes.io/projected/afa294db-fb48-47cd-84a8-2260c93aca2d-kube-api-access-j4xpt\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:20.601116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.601084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:20.638143 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.638108 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzr5\" (UniqueName: \"kubernetes.io/projected/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-kube-api-access-ztzr5\") pod \"odh-model-controller-7998c67dc4-v6xnk\" (UID: \"c2b7dc31-4af2-4db3-afd3-4fd6b644084f\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:20.638363 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.638185 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-cert\") pod \"odh-model-controller-7998c67dc4-v6xnk\" (UID: \"c2b7dc31-4af2-4db3-afd3-4fd6b644084f\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:20.638363 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:20.638316 2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 04 16:06:20.638474 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:20.638395 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-cert podName:c2b7dc31-4af2-4db3-afd3-4fd6b644084f nodeName:}" failed. No retries permitted until 2026-08-04 16:06:21.13837675 +0000 UTC m=+590.231976982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-cert") pod "odh-model-controller-7998c67dc4-v6xnk" (UID: "c2b7dc31-4af2-4db3-afd3-4fd6b644084f") : secret "odh-model-controller-webhook-cert" not found Aug 04 16:06:20.679261 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.679224 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzr5\" (UniqueName: \"kubernetes.io/projected/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-kube-api-access-ztzr5\") pod \"odh-model-controller-7998c67dc4-v6xnk\" (UID: \"c2b7dc31-4af2-4db3-afd3-4fd6b644084f\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:20.738124 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.738092 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:20.895364 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:20.895322 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:21.040967 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.040916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa294db-fb48-47cd-84a8-2260c93aca2d-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:21.043478 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.043457 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afa294db-fb48-47cd-84a8-2260c93aca2d-tls-certs\") pod \"kube-auth-proxy-78b5c5f4c4-kx7qj\" (UID: \"afa294db-fb48-47cd-84a8-2260c93aca2d\") " pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:21.141575 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.141530 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-cert\") pod \"odh-model-controller-7998c67dc4-v6xnk\" (UID: \"c2b7dc31-4af2-4db3-afd3-4fd6b644084f\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:21.144036 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.144009 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2b7dc31-4af2-4db3-afd3-4fd6b644084f-cert\") pod \"odh-model-controller-7998c67dc4-v6xnk\" (UID: \"c2b7dc31-4af2-4db3-afd3-4fd6b644084f\") " pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:21.336485 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.336446 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" Aug 04 16:06:21.415362 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.415324 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:21.514601 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.514563 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj"] Aug 04 16:06:21.515625 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:06:21.515595 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa294db_fb48_47cd_84a8_2260c93aca2d.slice/crio-67e334a7a437b8091f7df6ff0dad74ad74440139137f1a17a8b999276182284a WatchSource:0}: Error finding container 67e334a7a437b8091f7df6ff0dad74ad74440139137f1a17a8b999276182284a: Status 404 returned error can't find the container with id 67e334a7a437b8091f7df6ff0dad74ad74440139137f1a17a8b999276182284a Aug 04 16:06:21.573136 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.573106 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-v6xnk"] Aug 04 16:06:21.576008 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:06:21.575980 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b7dc31_4af2_4db3_afd3_4fd6b644084f.slice/crio-b77b11560b2e51e14a0c1be8460ed779d3f8de6c9e93647ed047501f0d4698bc WatchSource:0}: Error finding container b77b11560b2e51e14a0c1be8460ed779d3f8de6c9e93647ed047501f0d4698bc: Status 404 returned error can't find the container with id b77b11560b2e51e14a0c1be8460ed779d3f8de6c9e93647ed047501f0d4698bc Aug 04 16:06:21.801066 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.800969 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" event={"ID":"c2b7dc31-4af2-4db3-afd3-4fd6b644084f","Type":"ContainerStarted","Data":"b77b11560b2e51e14a0c1be8460ed779d3f8de6c9e93647ed047501f0d4698bc"} Aug 04 16:06:21.802127 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.802107 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" event={"ID":"afa294db-fb48-47cd-84a8-2260c93aca2d","Type":"ContainerStarted","Data":"67e334a7a437b8091f7df6ff0dad74ad74440139137f1a17a8b999276182284a"} Aug 04 16:06:21.882127 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.882089 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-b6jwv"] Aug 04 16:06:21.886907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.886882 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:21.893337 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.893307 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 04 16:06:21.901079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.901052 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-b6jwv"] Aug 04 16:06:21.951672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.951635 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76d46f0-0b35-43bd-b210-353a1f617907-tls-certs\") pod \"model-serving-api-54d89bd5fc-b6jwv\" (UID: \"b76d46f0-0b35-43bd-b210-353a1f617907\") " pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:21.951852 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:21.951704 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhvl\" (UniqueName: \"kubernetes.io/projected/b76d46f0-0b35-43bd-b210-353a1f617907-kube-api-access-blhvl\") pod \"model-serving-api-54d89bd5fc-b6jwv\" (UID: \"b76d46f0-0b35-43bd-b210-353a1f617907\") " pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:22.052790 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.052693 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhvl\" (UniqueName: \"kubernetes.io/projected/b76d46f0-0b35-43bd-b210-353a1f617907-kube-api-access-blhvl\") pod \"model-serving-api-54d89bd5fc-b6jwv\" (UID: \"b76d46f0-0b35-43bd-b210-353a1f617907\") " pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:22.052962 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.052807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76d46f0-0b35-43bd-b210-353a1f617907-tls-certs\") pod \"model-serving-api-54d89bd5fc-b6jwv\" (UID: \"b76d46f0-0b35-43bd-b210-353a1f617907\") " pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:22.053026 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:22.052957 2571 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Aug 04 16:06:22.053076 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:22.053035 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b76d46f0-0b35-43bd-b210-353a1f617907-tls-certs podName:b76d46f0-0b35-43bd-b210-353a1f617907 nodeName:}" failed. No retries permitted until 2026-08-04 16:06:22.55301045 +0000 UTC m=+591.646610677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b76d46f0-0b35-43bd-b210-353a1f617907-tls-certs") pod "model-serving-api-54d89bd5fc-b6jwv" (UID: "b76d46f0-0b35-43bd-b210-353a1f617907") : secret "model-serving-api-tls" not found Aug 04 16:06:22.067263 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.067230 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhvl\" (UniqueName: \"kubernetes.io/projected/b76d46f0-0b35-43bd-b210-353a1f617907-kube-api-access-blhvl\") pod \"model-serving-api-54d89bd5fc-b6jwv\" (UID: \"b76d46f0-0b35-43bd-b210-353a1f617907\") " pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:22.559392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.559347 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76d46f0-0b35-43bd-b210-353a1f617907-tls-certs\") pod \"model-serving-api-54d89bd5fc-b6jwv\" (UID: \"b76d46f0-0b35-43bd-b210-353a1f617907\") " pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:22.574039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.574006 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76d46f0-0b35-43bd-b210-353a1f617907-tls-certs\") pod \"model-serving-api-54d89bd5fc-b6jwv\" (UID: \"b76d46f0-0b35-43bd-b210-353a1f617907\") " pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:22.761488 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.761403 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:22.797499 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.797461 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:22.854007 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:22.853968 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:23.009593 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:23.009560 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-b6jwv"] Aug 04 16:06:23.012434 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:06:23.012397 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76d46f0_0b35_43bd_b210_353a1f617907.slice/crio-96cd75b4c3949c468f50ad5e1294bfac02a146c1776010664c9f55a489cc411b WatchSource:0}: Error finding container 96cd75b4c3949c468f50ad5e1294bfac02a146c1776010664c9f55a489cc411b: Status 404 returned error can't find the container with id 96cd75b4c3949c468f50ad5e1294bfac02a146c1776010664c9f55a489cc411b Aug 04 16:06:23.793826 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:23.793785 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:23.816944 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:23.816865 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" event={"ID":"b76d46f0-0b35-43bd-b210-353a1f617907","Type":"ContainerStarted","Data":"96cd75b4c3949c468f50ad5e1294bfac02a146c1776010664c9f55a489cc411b"} Aug 04 16:06:26.835979 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:26.835942 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" event={"ID":"c2b7dc31-4af2-4db3-afd3-4fd6b644084f","Type":"ContainerStarted","Data":"6519cd63dfafa9a2d2179357ce734b125a1952a03e161c95efd1e69688ff54a6"} Aug 04 16:06:26.836449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:26.836043 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:26.837682 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:26.837658 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" event={"ID":"afa294db-fb48-47cd-84a8-2260c93aca2d","Type":"ContainerStarted","Data":"d390e2701e73f156426a22d546d43af8e80df5007f03cc7bf21bb81d567305e4"} Aug 04 16:06:26.855876 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:26.855819 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" podStartSLOduration=2.541194895 podStartE2EDuration="6.855803537s" totalImagesPullingTime="4.314608642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:26.854548642 +0000 UTC m=+595.948148901" watchObservedRunningTime="2026-08-04 16:06:26.855803537 +0000 UTC m=+595.949403784" Aug 04 16:06:26.875357 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:26.875297 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78b5c5f4c4-kx7qj" podStartSLOduration=2.542389944 podStartE2EDuration="6.875279456s" totalImagesPullingTime="4.332889512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:26.872942396 +0000 UTC m=+595.966542645" watchObservedRunningTime="2026-08-04 16:06:26.875279456 +0000 UTC m=+595.968879705" Aug 04 16:06:27.844964 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:27.844925 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" event={"ID":"b76d46f0-0b35-43bd-b210-353a1f617907","Type":"ContainerStarted","Data":"8bf59a062d37ec3fbe3c782e3db92217653c1595d1f64889033e5bd3c10f4600"} Aug 04 16:06:27.845490 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:27.845393 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:27.892352 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:27.892286 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" podStartSLOduration=2.477080489 podStartE2EDuration="6.892268188s" totalImagesPullingTime="4.415187699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:27.890879602 +0000 UTC m=+596.984479851" watchObservedRunningTime="2026-08-04 16:06:27.892268188 +0000 UTC m=+596.985868493" Aug 04 16:06:28.420781 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:28.420723 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:28.534577 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:28.534542 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:29.147694 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.147659 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk"] Aug 04 16:06:29.151372 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.151353 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.154213 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.154177 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 16:06:29.154412 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.154198 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 16:06:29.154412 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.154177 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 16:06:29.154412 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.154245 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 16:06:29.154412 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.154286 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4tx97\"" Aug 04 16:06:29.165919 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.165879 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk"] Aug 04 16:06:29.227239 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.227162 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.227449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.227265 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.227449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.227310 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.227449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.227360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.227449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.227393 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4380e24f-b3fc-4d52-a094-c4de038657fe-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.227449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.227428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.227651 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.227456 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6mb\" (UniqueName: \"kubernetes.io/projected/4380e24f-b3fc-4d52-a094-c4de038657fe-kube-api-access-ww6mb\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.328767 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.328726 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6mb\" (UniqueName: \"kubernetes.io/projected/4380e24f-b3fc-4d52-a094-c4de038657fe-kube-api-access-ww6mb\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.328973 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.328800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.328973 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.328854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.328973 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.328904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.328973 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.328966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.329184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.329000 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4380e24f-b3fc-4d52-a094-c4de038657fe-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.329184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.329041 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.329794 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.329767 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.331506 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.331475 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4380e24f-b3fc-4d52-a094-c4de038657fe-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.331803 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.331664 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.331977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.331818 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.332049 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.332024 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4380e24f-b3fc-4d52-a094-c4de038657fe-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.350274 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.350238 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4380e24f-b3fc-4d52-a094-c4de038657fe-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.353313 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.353287 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6mb\" (UniqueName: \"kubernetes.io/projected/4380e24f-b3fc-4d52-a094-c4de038657fe-kube-api-access-ww6mb\") pod \"istiod-openshift-gateway-5677ccb49b-rjgkk\" (UID: \"4380e24f-b3fc-4d52-a094-c4de038657fe\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.452482 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.452385 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:29.461335 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.461296 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:29.684898 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.684860 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk"] Aug 04 16:06:29.688110 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:06:29.688076 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4380e24f_b3fc_4d52_a094_c4de038657fe.slice/crio-20e6e3466b245de839fc4dfd2e6aa1926ef49144945c22ee217a0e8fa5e7d45f WatchSource:0}: Error finding container 20e6e3466b245de839fc4dfd2e6aa1926ef49144945c22ee217a0e8fa5e7d45f: Status 404 returned error can't find the container with id 20e6e3466b245de839fc4dfd2e6aa1926ef49144945c22ee217a0e8fa5e7d45f Aug 04 16:06:29.690087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.690054 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:29.804725 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.804692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:29.857102 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:29.857067 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" event={"ID":"4380e24f-b3fc-4d52-a094-c4de038657fe","Type":"ContainerStarted","Data":"20e6e3466b245de839fc4dfd2e6aa1926ef49144945c22ee217a0e8fa5e7d45f"} Aug 04 16:06:32.140281 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.140183 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:06:32.140600 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.140314 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:06:32.397101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.396949 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:32.507013 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.506978 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b"] Aug 04 16:06:32.514893 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.514864 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:32.517981 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.517946 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:32.517981 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.517969 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hxkms\"" Aug 04 16:06:32.517981 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.517981 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Aug 04 16:06:32.522760 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.522706 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b"] Aug 04 16:06:32.560987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.560949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5ck\" (UniqueName: \"kubernetes.io/projected/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-kube-api-access-8p5ck\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:32.561198 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.561034 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:32.662056 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.661967 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:32.662056 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.662039 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5ck\" (UniqueName: \"kubernetes.io/projected/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-kube-api-access-8p5ck\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:32.662276 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:32.662121 2571 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Aug 04 16:06:32.662276 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:32.662190 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert podName:6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef nodeName:}" failed. No retries permitted until 2026-08-04 16:06:33.162172641 +0000 UTC m=+602.255772867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert") pod "llmisvc-controller-manager-777c6bd7d5-kv24b" (UID: "6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef") : secret "llmisvc-webhook-server-cert" not found Aug 04 16:06:32.686868 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.686835 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5ck\" (UniqueName: \"kubernetes.io/projected/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-kube-api-access-8p5ck\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:32.879912 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.879740 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" event={"ID":"4380e24f-b3fc-4d52-a094-c4de038657fe","Type":"ContainerStarted","Data":"c09d2005c8385f08bcfc3efe4fe0fff6ecd12ec32e140ea8a192affe264115ff"} Aug 04 16:06:32.879912 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.879844 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:32.926222 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:32.926053 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" podStartSLOduration=1.4765479670000001 podStartE2EDuration="3.926032461s" totalImagesPullingTime="2.449484494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:32.925670753 +0000 UTC m=+602.019271001" watchObservedRunningTime="2026-08-04 16:06:32.926032461 +0000 UTC m=+602.019632710" Aug 04 16:06:33.167489 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:33.167446 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:33.167881 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:33.167611 2571 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Aug 04 16:06:33.167881 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:06:33.167698 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert podName:6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef nodeName:}" failed. No retries permitted until 2026-08-04 16:06:34.167682031 +0000 UTC m=+603.261282261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert") pod "llmisvc-controller-manager-777c6bd7d5-kv24b" (UID: "6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef") : secret "llmisvc-webhook-server-cert" not found Aug 04 16:06:33.402151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:33.402117 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:33.746599 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:33.746508 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:33.886735 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:33.886706 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-rjgkk" Aug 04 16:06:34.178701 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:34.178655 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:34.181152 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:34.181118 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kv24b\" (UID: \"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:34.328277 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:34.328233 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:34.492707 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:34.492682 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b"] Aug 04 16:06:34.498485 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:06:34.495360 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b0e5f25_ce3f_4b16_8462_dbeb6773a6ef.slice/crio-0604648c40784a550131f1bbbf2d40c27f2d3e62068a263671bd1c25c7f5f8ed WatchSource:0}: Error finding container 0604648c40784a550131f1bbbf2d40c27f2d3e62068a263671bd1c25c7f5f8ed: Status 404 returned error can't find the container with id 0604648c40784a550131f1bbbf2d40c27f2d3e62068a263671bd1c25c7f5f8ed Aug 04 16:06:34.890370 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:34.890332 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" event={"ID":"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef","Type":"ContainerStarted","Data":"0604648c40784a550131f1bbbf2d40c27f2d3e62068a263671bd1c25c7f5f8ed"} Aug 04 16:06:35.355840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:35.355808 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:37.847889 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:37.847851 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-v6xnk" Aug 04 16:06:38.918110 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:38.918070 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" event={"ID":"6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef","Type":"ContainerStarted","Data":"05bc98402683ae1be4e486b1dce2e60eca039acfbb977112c797e134786f5d6f"} Aug 04 16:06:38.918529 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:38.918158 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:06:38.940932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:38.940870 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" podStartSLOduration=3.270857627 podStartE2EDuration="6.940852448s" totalImagesPullingTime="3.669994821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:38.938106014 +0000 UTC m=+608.031706262" watchObservedRunningTime="2026-08-04 16:06:38.940852448 +0000 UTC m=+608.034452696" Aug 04 16:06:39.861837 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:39.861804 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-b6jwv" Aug 04 16:06:44.481263 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:06:44.481226 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:09.925294 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:09.925193 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kv24b" Aug 04 16:07:21.478239 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:21.478174 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:23.837276 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:23.837240 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:25.047369 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.047326 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw"] Aug 04 16:07:25.055631 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.055597 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.059170 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.058945 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Aug 04 16:07:25.059170 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.059150 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nt86t\"" Aug 04 16:07:25.059627 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.059600 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Aug 04 16:07:25.060258 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.060238 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw"] Aug 04 16:07:25.149782 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.149749 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.149970 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.149793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.149970 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.149820 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqmz\" (UniqueName: \"kubernetes.io/projected/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-kube-api-access-rjqmz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.150648 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.150628 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j"] Aug 04 16:07:25.153319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.153303 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.164098 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.164069 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j"] Aug 04 16:07:25.250838 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.250787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.250838 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.250843 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmlg6\" (UniqueName: \"kubernetes.io/projected/00818e8c-aa1c-4130-9d2f-4f395531b673-kube-api-access-fmlg6\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.251092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.250963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqmz\" (UniqueName: \"kubernetes.io/projected/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-kube-api-access-rjqmz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.251092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.251007 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.251092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.251090 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.251287 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.251187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.251287 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.251269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.251519 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.251495 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.255550 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.255528 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq"] Aug 04 16:07:25.258240 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.258225 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.261725 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.261694 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqmz\" (UniqueName: \"kubernetes.io/projected/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-kube-api-access-rjqmz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.271416 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.271387 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq"] Aug 04 16:07:25.352358 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352314 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmlg6\" (UniqueName: \"kubernetes.io/projected/00818e8c-aa1c-4130-9d2f-4f395531b673-kube-api-access-fmlg6\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.352543 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.352543 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.352543 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.352543 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352495 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crvnc\" (UniqueName: \"kubernetes.io/projected/be745497-8743-403f-a78a-c31e79417cb1-kube-api-access-crvnc\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.352543 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.352865 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352842 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.352865 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.352858 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.363577 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.363550 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc"] Aug 04 16:07:25.364919 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.364895 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmlg6\" (UniqueName: \"kubernetes.io/projected/00818e8c-aa1c-4130-9d2f-4f395531b673-kube-api-access-fmlg6\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.366340 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.366324 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.366666 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.366645 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:25.380482 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.380173 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc"] Aug 04 16:07:25.453551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.453512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.453551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.453554 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.453751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.453586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crvnc\" (UniqueName: \"kubernetes.io/projected/be745497-8743-403f-a78a-c31e79417cb1-kube-api-access-crvnc\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.453751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.453672 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.453751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.453704 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.453751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.453719 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts29m\" (UniqueName: \"kubernetes.io/projected/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-kube-api-access-ts29m\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.454080 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.454064 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.454130 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.454111 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.462751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.462723 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crvnc\" (UniqueName: \"kubernetes.io/projected/be745497-8743-403f-a78a-c31e79417cb1-kube-api-access-crvnc\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.462915 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.462866 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:25.511527 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.511500 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw"] Aug 04 16:07:25.513966 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:07:25.513936 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374df0cf_9bb7_4c5b_a3c1_8b661316d52b.slice/crio-81fd730f56db5c13c0d49ea0f4f49d3ab5748d16c18a047eff76727d2c4bbf1f WatchSource:0}: Error finding container 81fd730f56db5c13c0d49ea0f4f49d3ab5748d16c18a047eff76727d2c4bbf1f: Status 404 returned error can't find the container with id 81fd730f56db5c13c0d49ea0f4f49d3ab5748d16c18a047eff76727d2c4bbf1f Aug 04 16:07:25.554450 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.554403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts29m\" (UniqueName: \"kubernetes.io/projected/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-kube-api-access-ts29m\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.554733 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.554470 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.554889 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.554751 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.555316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.555267 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.555724 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.555577 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.564278 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.564252 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts29m\" (UniqueName: \"kubernetes.io/projected/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-kube-api-access-ts29m\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.576694 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.576431 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:25.608685 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.608592 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j"] Aug 04 16:07:25.612044 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:07:25.612012 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00818e8c_aa1c_4130_9d2f_4f395531b673.slice/crio-346af8af2431fece5f903b96f67b7783a3a7368dbd42847a756847c8dc661d52 WatchSource:0}: Error finding container 346af8af2431fece5f903b96f67b7783a3a7368dbd42847a756847c8dc661d52: Status 404 returned error can't find the container with id 346af8af2431fece5f903b96f67b7783a3a7368dbd42847a756847c8dc661d52 Aug 04 16:07:25.699128 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.699095 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:25.726855 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.726823 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq"] Aug 04 16:07:25.727581 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:07:25.727554 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe745497_8743_403f_a78a_c31e79417cb1.slice/crio-e079662c3d37b545a31dd8a7b0138e9a47adbe64e53f99f45780b3a8040498d3 WatchSource:0}: Error finding container e079662c3d37b545a31dd8a7b0138e9a47adbe64e53f99f45780b3a8040498d3: Status 404 returned error can't find the container with id e079662c3d37b545a31dd8a7b0138e9a47adbe64e53f99f45780b3a8040498d3 Aug 04 16:07:25.849232 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:25.849174 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc"] Aug 04 16:07:25.850448 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:07:25.850411 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315ae7cd_2ec3_4d03_837f_be320e7f3ff5.slice/crio-3125cdb25ae1401dc03f7db5efac685330101dfdc934e4d4cc2fa48cc18c1c08 WatchSource:0}: Error finding container 3125cdb25ae1401dc03f7db5efac685330101dfdc934e4d4cc2fa48cc18c1c08: Status 404 returned error can't find the container with id 3125cdb25ae1401dc03f7db5efac685330101dfdc934e4d4cc2fa48cc18c1c08 Aug 04 16:07:26.183864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.183765 2571 generic.go:356] "Generic (PLEG): container finished" podID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerID="4016791ec5f12a6440507ab731cee001fc9765c84e5524a0c8cca5da99e5546a" exitCode=0 Aug 04 16:07:26.184309 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.183852 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" event={"ID":"315ae7cd-2ec3-4d03-837f-be320e7f3ff5","Type":"ContainerDied","Data":"4016791ec5f12a6440507ab731cee001fc9765c84e5524a0c8cca5da99e5546a"} Aug 04 16:07:26.184309 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.183908 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" event={"ID":"315ae7cd-2ec3-4d03-837f-be320e7f3ff5","Type":"ContainerStarted","Data":"3125cdb25ae1401dc03f7db5efac685330101dfdc934e4d4cc2fa48cc18c1c08"} Aug 04 16:07:26.185678 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.185655 2571 generic.go:356] "Generic (PLEG): container finished" podID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerID="fc160a141a4273c59da2bbab5cb85670af2db6790f8f9333ea8047a6494bf1c6" exitCode=0 Aug 04 16:07:26.185768 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.185743 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" event={"ID":"374df0cf-9bb7-4c5b-a3c1-8b661316d52b","Type":"ContainerDied","Data":"fc160a141a4273c59da2bbab5cb85670af2db6790f8f9333ea8047a6494bf1c6"} Aug 04 16:07:26.185832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.185788 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" event={"ID":"374df0cf-9bb7-4c5b-a3c1-8b661316d52b","Type":"ContainerStarted","Data":"81fd730f56db5c13c0d49ea0f4f49d3ab5748d16c18a047eff76727d2c4bbf1f"} Aug 04 16:07:26.187538 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.187514 2571 generic.go:356] "Generic (PLEG): container finished" podID="be745497-8743-403f-a78a-c31e79417cb1" containerID="e569743ad21d9ed2a65f36c1b476a3a1237ecd9170dcef06b014a385a7692cdd" exitCode=0 Aug 04 16:07:26.187638 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.187590 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" event={"ID":"be745497-8743-403f-a78a-c31e79417cb1","Type":"ContainerDied","Data":"e569743ad21d9ed2a65f36c1b476a3a1237ecd9170dcef06b014a385a7692cdd"} Aug 04 16:07:26.187708 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.187643 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" event={"ID":"be745497-8743-403f-a78a-c31e79417cb1","Type":"ContainerStarted","Data":"e079662c3d37b545a31dd8a7b0138e9a47adbe64e53f99f45780b3a8040498d3"} Aug 04 16:07:26.189364 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.189343 2571 generic.go:356] "Generic (PLEG): container finished" podID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerID="24fe77be72e3d27aed39981eaa6210a3ee30d13db6eabf46da1c385b1fdc84ff" exitCode=0 Aug 04 16:07:26.189466 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.189371 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" event={"ID":"00818e8c-aa1c-4130-9d2f-4f395531b673","Type":"ContainerDied","Data":"24fe77be72e3d27aed39981eaa6210a3ee30d13db6eabf46da1c385b1fdc84ff"} Aug 04 16:07:26.189466 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:26.189399 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" event={"ID":"00818e8c-aa1c-4130-9d2f-4f395531b673","Type":"ContainerStarted","Data":"346af8af2431fece5f903b96f67b7783a3a7368dbd42847a756847c8dc661d52"} Aug 04 16:07:27.196550 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:27.196516 2571 generic.go:356] "Generic (PLEG): container finished" podID="be745497-8743-403f-a78a-c31e79417cb1" containerID="2b3907dd65ead34d6893cdaaeb813c64d9c3694b384134bfdd3c8543026e44e5" exitCode=0 Aug 04 16:07:27.197034 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:27.196562 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" event={"ID":"be745497-8743-403f-a78a-c31e79417cb1","Type":"ContainerDied","Data":"2b3907dd65ead34d6893cdaaeb813c64d9c3694b384134bfdd3c8543026e44e5"} Aug 04 16:07:27.198593 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:27.198572 2571 generic.go:356] "Generic (PLEG): container finished" podID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerID="1adbe60c6ed290340b519a0de8b9a0c540bad2aa7d02d44318532db102066748" exitCode=0 Aug 04 16:07:27.198686 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:27.198620 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" event={"ID":"374df0cf-9bb7-4c5b-a3c1-8b661316d52b","Type":"ContainerDied","Data":"1adbe60c6ed290340b519a0de8b9a0c540bad2aa7d02d44318532db102066748"} Aug 04 16:07:28.206798 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:28.206761 2571 generic.go:356] "Generic (PLEG): container finished" podID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerID="b1f2af66bcb02c5cd2a31f67751127ce7e078ec8b13f5021c7f8ef2ee4687093" exitCode=0 Aug 04 16:07:28.207198 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:28.206855 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" event={"ID":"374df0cf-9bb7-4c5b-a3c1-8b661316d52b","Type":"ContainerDied","Data":"b1f2af66bcb02c5cd2a31f67751127ce7e078ec8b13f5021c7f8ef2ee4687093"} Aug 04 16:07:28.209015 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:28.208992 2571 generic.go:356] "Generic (PLEG): container finished" podID="be745497-8743-403f-a78a-c31e79417cb1" containerID="a75e25eb4205b496c7c0d9842a4291d9c957a08586affb12b23d7248c0301243" exitCode=0 Aug 04 16:07:28.209143 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:28.209057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" event={"ID":"be745497-8743-403f-a78a-c31e79417cb1","Type":"ContainerDied","Data":"a75e25eb4205b496c7c0d9842a4291d9c957a08586affb12b23d7248c0301243"} Aug 04 16:07:29.216758 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.216726 2571 generic.go:356] "Generic (PLEG): container finished" podID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerID="817c0ed4996647e1bc63ac60ff55014fdaf44c91e02ed122b3f68481ba39883c" exitCode=0 Aug 04 16:07:29.217176 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.216813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" event={"ID":"00818e8c-aa1c-4130-9d2f-4f395531b673","Type":"ContainerDied","Data":"817c0ed4996647e1bc63ac60ff55014fdaf44c91e02ed122b3f68481ba39883c"} Aug 04 16:07:29.218930 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.218910 2571 generic.go:356] "Generic (PLEG): container finished" podID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerID="274468eb1ac97e612117fd3024ee780e8cba6227dfc863bf088fc710263b72c2" exitCode=0 Aug 04 16:07:29.219009 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.218987 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" event={"ID":"315ae7cd-2ec3-4d03-837f-be320e7f3ff5","Type":"ContainerDied","Data":"274468eb1ac97e612117fd3024ee780e8cba6227dfc863bf088fc710263b72c2"} Aug 04 16:07:29.365821 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.365790 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:29.468521 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.468456 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:29.490978 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.490947 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-util\") pod \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " Aug 04 16:07:29.491134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.490985 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjqmz\" (UniqueName: \"kubernetes.io/projected/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-kube-api-access-rjqmz\") pod \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " Aug 04 16:07:29.491134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.491096 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-bundle\") pod \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\" (UID: \"374df0cf-9bb7-4c5b-a3c1-8b661316d52b\") " Aug 04 16:07:29.491723 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.491687 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-bundle" (OuterVolumeSpecName: "bundle") pod "374df0cf-9bb7-4c5b-a3c1-8b661316d52b" (UID: "374df0cf-9bb7-4c5b-a3c1-8b661316d52b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:29.493091 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.493070 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-kube-api-access-rjqmz" (OuterVolumeSpecName: "kube-api-access-rjqmz") pod "374df0cf-9bb7-4c5b-a3c1-8b661316d52b" (UID: "374df0cf-9bb7-4c5b-a3c1-8b661316d52b"). InnerVolumeSpecName "kube-api-access-rjqmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:07:29.496226 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.496190 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-util" (OuterVolumeSpecName: "util") pod "374df0cf-9bb7-4c5b-a3c1-8b661316d52b" (UID: "374df0cf-9bb7-4c5b-a3c1-8b661316d52b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:29.592193 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.592161 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crvnc\" (UniqueName: \"kubernetes.io/projected/be745497-8743-403f-a78a-c31e79417cb1-kube-api-access-crvnc\") pod \"be745497-8743-403f-a78a-c31e79417cb1\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " Aug 04 16:07:29.592403 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.592259 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-util\") pod \"be745497-8743-403f-a78a-c31e79417cb1\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " Aug 04 16:07:29.592403 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.592349 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-bundle\") pod \"be745497-8743-403f-a78a-c31e79417cb1\" (UID: \"be745497-8743-403f-a78a-c31e79417cb1\") " Aug 04 16:07:29.592542 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.592532 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:29.592596 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.592542 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:29.592596 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.592551 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjqmz\" (UniqueName: \"kubernetes.io/projected/374df0cf-9bb7-4c5b-a3c1-8b661316d52b-kube-api-access-rjqmz\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:29.592787 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.592757 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-bundle" (OuterVolumeSpecName: "bundle") pod "be745497-8743-403f-a78a-c31e79417cb1" (UID: "be745497-8743-403f-a78a-c31e79417cb1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:29.594293 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.594268 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be745497-8743-403f-a78a-c31e79417cb1-kube-api-access-crvnc" (OuterVolumeSpecName: "kube-api-access-crvnc") pod "be745497-8743-403f-a78a-c31e79417cb1" (UID: "be745497-8743-403f-a78a-c31e79417cb1"). InnerVolumeSpecName "kube-api-access-crvnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:07:29.597756 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.597718 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-util" (OuterVolumeSpecName: "util") pod "be745497-8743-403f-a78a-c31e79417cb1" (UID: "be745497-8743-403f-a78a-c31e79417cb1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:29.693073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.693037 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:29.693073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.693067 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crvnc\" (UniqueName: \"kubernetes.io/projected/be745497-8743-403f-a78a-c31e79417cb1-kube-api-access-crvnc\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:29.693073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:29.693076 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be745497-8743-403f-a78a-c31e79417cb1-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:30.230029 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.229991 2571 generic.go:356] "Generic (PLEG): container finished" podID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerID="e9b3d9d297026986788e04e8c7fa3d80c25b2ec0b4e182afe15fb85f3fca77bb" exitCode=0 Aug 04 16:07:30.230453 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.230035 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" event={"ID":"315ae7cd-2ec3-4d03-837f-be320e7f3ff5","Type":"ContainerDied","Data":"e9b3d9d297026986788e04e8c7fa3d80c25b2ec0b4e182afe15fb85f3fca77bb"} Aug 04 16:07:30.232149 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.232131 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" Aug 04 16:07:30.232316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.232160 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cpfrxw" event={"ID":"374df0cf-9bb7-4c5b-a3c1-8b661316d52b","Type":"ContainerDied","Data":"81fd730f56db5c13c0d49ea0f4f49d3ab5748d16c18a047eff76727d2c4bbf1f"} Aug 04 16:07:30.232316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.232190 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81fd730f56db5c13c0d49ea0f4f49d3ab5748d16c18a047eff76727d2c4bbf1f" Aug 04 16:07:30.234049 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.234031 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" Aug 04 16:07:30.234154 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.234048 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0npgvq" event={"ID":"be745497-8743-403f-a78a-c31e79417cb1","Type":"ContainerDied","Data":"e079662c3d37b545a31dd8a7b0138e9a47adbe64e53f99f45780b3a8040498d3"} Aug 04 16:07:30.234154 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.234084 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e079662c3d37b545a31dd8a7b0138e9a47adbe64e53f99f45780b3a8040498d3" Aug 04 16:07:30.236432 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.236409 2571 generic.go:356] "Generic (PLEG): container finished" podID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerID="ad26370f2f61d6b1b4b77d33d1ff5e8441d6fbe9abca78a3c2e9d59c5c2ea241" exitCode=0 Aug 04 16:07:30.236551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:30.236486 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" event={"ID":"00818e8c-aa1c-4130-9d2f-4f395531b673","Type":"ContainerDied","Data":"ad26370f2f61d6b1b4b77d33d1ff5e8441d6fbe9abca78a3c2e9d59c5c2ea241"} Aug 04 16:07:31.337698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.337577 2571 scope.go:117] "RemoveContainer" containerID="7364b054451f90c5c40f458e4ee91f351e9da2da51b06a201c34084410c709cb" Aug 04 16:07:31.348176 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.348151 2571 scope.go:117] "RemoveContainer" containerID="f180e0ac51ac54e11125e6184c4bbfe3d2fc437c0ab2cfdd51649de110f96ee4" Aug 04 16:07:31.357621 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.357567 2571 scope.go:117] "RemoveContainer" containerID="d3f34c278ce09405c535410b935a9e4a7f6d8e038cf0a275f4530a5c455ede63" Aug 04 16:07:31.368837 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.368811 2571 scope.go:117] "RemoveContainer" containerID="19a4dfcb94e0dd595f4e13d9add4e2326ef4f877e0946199d9d298d9d29af652" Aug 04 16:07:31.387953 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.387934 2571 scope.go:117] "RemoveContainer" containerID="71df0d2e5c6e8c9884f32b5943bee8cff582f9690dfc841607fb6d503f7e9022" Aug 04 16:07:31.405894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.405869 2571 scope.go:117] "RemoveContainer" containerID="3a7501250e34b2f9ae871bbbd11d342057f6d18a82b87b9af04d75d98eec5561" Aug 04 16:07:31.410457 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.410434 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:31.413421 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.413404 2571 scope.go:117] "RemoveContainer" containerID="834a23d3682d11010fd5e0865ddb181a342af3fc2388cfee5131bc4de5daa9df" Aug 04 16:07:31.415002 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.414979 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:31.511339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.511253 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts29m\" (UniqueName: \"kubernetes.io/projected/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-kube-api-access-ts29m\") pod \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " Aug 04 16:07:31.511339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.511327 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-util\") pod \"00818e8c-aa1c-4130-9d2f-4f395531b673\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " Aug 04 16:07:31.511551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.511365 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-bundle\") pod \"00818e8c-aa1c-4130-9d2f-4f395531b673\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " Aug 04 16:07:31.511551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.511414 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-util\") pod \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " Aug 04 16:07:31.511551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.511499 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmlg6\" (UniqueName: \"kubernetes.io/projected/00818e8c-aa1c-4130-9d2f-4f395531b673-kube-api-access-fmlg6\") pod \"00818e8c-aa1c-4130-9d2f-4f395531b673\" (UID: \"00818e8c-aa1c-4130-9d2f-4f395531b673\") " Aug 04 16:07:31.511551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.511529 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-bundle\") pod \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\" (UID: \"315ae7cd-2ec3-4d03-837f-be320e7f3ff5\") " Aug 04 16:07:31.512257 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.512222 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-bundle" (OuterVolumeSpecName: "bundle") pod "00818e8c-aa1c-4130-9d2f-4f395531b673" (UID: "00818e8c-aa1c-4130-9d2f-4f395531b673"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:31.512478 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.512449 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-bundle" (OuterVolumeSpecName: "bundle") pod "315ae7cd-2ec3-4d03-837f-be320e7f3ff5" (UID: "315ae7cd-2ec3-4d03-837f-be320e7f3ff5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:31.513657 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.513634 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00818e8c-aa1c-4130-9d2f-4f395531b673-kube-api-access-fmlg6" (OuterVolumeSpecName: "kube-api-access-fmlg6") pod "00818e8c-aa1c-4130-9d2f-4f395531b673" (UID: "00818e8c-aa1c-4130-9d2f-4f395531b673"). InnerVolumeSpecName "kube-api-access-fmlg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:07:31.513763 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.513664 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-kube-api-access-ts29m" (OuterVolumeSpecName: "kube-api-access-ts29m") pod "315ae7cd-2ec3-4d03-837f-be320e7f3ff5" (UID: "315ae7cd-2ec3-4d03-837f-be320e7f3ff5"). InnerVolumeSpecName "kube-api-access-ts29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:07:31.516197 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.516172 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-util" (OuterVolumeSpecName: "util") pod "00818e8c-aa1c-4130-9d2f-4f395531b673" (UID: "00818e8c-aa1c-4130-9d2f-4f395531b673"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:31.516776 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.516757 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-util" (OuterVolumeSpecName: "util") pod "315ae7cd-2ec3-4d03-837f-be320e7f3ff5" (UID: "315ae7cd-2ec3-4d03-837f-be320e7f3ff5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:07:31.612805 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.612761 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmlg6\" (UniqueName: \"kubernetes.io/projected/00818e8c-aa1c-4130-9d2f-4f395531b673-kube-api-access-fmlg6\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:31.612805 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.612795 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:31.612805 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.612806 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts29m\" (UniqueName: \"kubernetes.io/projected/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-kube-api-access-ts29m\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:31.612805 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.612818 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:31.613082 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.612826 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00818e8c-aa1c-4130-9d2f-4f395531b673-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:31.613082 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:31.612834 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/315ae7cd-2ec3-4d03-837f-be320e7f3ff5-util\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:07:32.258553 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:32.258522 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" Aug 04 16:07:32.258714 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:32.258546 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75hpc" event={"ID":"315ae7cd-2ec3-4d03-837f-be320e7f3ff5","Type":"ContainerDied","Data":"3125cdb25ae1401dc03f7db5efac685330101dfdc934e4d4cc2fa48cc18c1c08"} Aug 04 16:07:32.258714 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:32.258596 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3125cdb25ae1401dc03f7db5efac685330101dfdc934e4d4cc2fa48cc18c1c08" Aug 04 16:07:32.260444 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:32.260423 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" Aug 04 16:07:32.260557 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:32.260454 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870536nq9j" event={"ID":"00818e8c-aa1c-4130-9d2f-4f395531b673","Type":"ContainerDied","Data":"346af8af2431fece5f903b96f67b7783a3a7368dbd42847a756847c8dc661d52"} Aug 04 16:07:32.260557 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:32.260486 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346af8af2431fece5f903b96f67b7783a3a7368dbd42847a756847c8dc661d52" Aug 04 16:07:36.690156 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:36.690125 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:37.514463 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:37.514428 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:38.348006 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:38.347973 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:41.231077 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:41.231045 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:41.357537 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:41.357500 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:41.755569 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:41.755532 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:42.701655 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:42.701616 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:42.814520 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:42.814479 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:43.886064 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886029 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-c8jf6"] Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886382 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerName="extract" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886396 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerName="extract" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886410 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerName="pull" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886415 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerName="pull" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886420 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerName="extract" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886426 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerName="extract" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886433 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerName="util" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886438 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerName="util" Aug 04 16:07:43.886443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886447 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerName="pull" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886452 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerName="pull" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886460 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerName="util" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886467 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerName="util" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886473 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be745497-8743-403f-a78a-c31e79417cb1" containerName="util" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886479 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="be745497-8743-403f-a78a-c31e79417cb1" containerName="util" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886484 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerName="pull" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886489 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerName="pull" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886495 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerName="util" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886500 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerName="util" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886507 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be745497-8743-403f-a78a-c31e79417cb1" containerName="extract" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886512 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="be745497-8743-403f-a78a-c31e79417cb1" containerName="extract" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886521 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be745497-8743-403f-a78a-c31e79417cb1" containerName="pull" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886526 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="be745497-8743-403f-a78a-c31e79417cb1" containerName="pull" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886531 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerName="extract" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886535 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerName="extract" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886600 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="374df0cf-9bb7-4c5b-a3c1-8b661316d52b" containerName="extract" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886609 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="315ae7cd-2ec3-4d03-837f-be320e7f3ff5" containerName="extract" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886616 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="be745497-8743-403f-a78a-c31e79417cb1" containerName="extract" Aug 04 16:07:43.886721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.886622 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="00818e8c-aa1c-4130-9d2f-4f395531b673" containerName="extract" Aug 04 16:07:43.891701 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.891681 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" Aug 04 16:07:43.897372 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.897352 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 16:07:43.897372 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.897359 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-cqzgd\"" Aug 04 16:07:43.897679 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.897663 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 16:07:43.907488 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.907466 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-c8jf6"] Aug 04 16:07:43.917764 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:43.917744 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9cb\" (UniqueName: \"kubernetes.io/projected/60059be3-f43e-4806-a420-c34245f4fc23-kube-api-access-wk9cb\") pod \"authorino-operator-68d88948d8-c8jf6\" (UID: \"60059be3-f43e-4806-a420-c34245f4fc23\") " pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" Aug 04 16:07:44.019009 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:44.018979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9cb\" (UniqueName: \"kubernetes.io/projected/60059be3-f43e-4806-a420-c34245f4fc23-kube-api-access-wk9cb\") pod \"authorino-operator-68d88948d8-c8jf6\" (UID: \"60059be3-f43e-4806-a420-c34245f4fc23\") " pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" Aug 04 16:07:44.034488 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:44.034463 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9cb\" (UniqueName: \"kubernetes.io/projected/60059be3-f43e-4806-a420-c34245f4fc23-kube-api-access-wk9cb\") pod \"authorino-operator-68d88948d8-c8jf6\" (UID: \"60059be3-f43e-4806-a420-c34245f4fc23\") " pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" Aug 04 16:07:44.201827 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:44.201742 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" Aug 04 16:07:44.342447 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:44.342418 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-c8jf6"] Aug 04 16:07:44.343263 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:07:44.343192 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60059be3_f43e_4806_a420_c34245f4fc23.slice/crio-f742aa811690146b28db8989429a6e7d6b295033151bd88877c0dc82857e7277 WatchSource:0}: Error finding container f742aa811690146b28db8989429a6e7d6b295033151bd88877c0dc82857e7277: Status 404 returned error can't find the container with id f742aa811690146b28db8989429a6e7d6b295033151bd88877c0dc82857e7277 Aug 04 16:07:44.560813 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:44.560782 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:44.733548 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:44.733515 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:45.333172 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:45.333138 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" event={"ID":"60059be3-f43e-4806-a420-c34245f4fc23","Type":"ContainerStarted","Data":"f742aa811690146b28db8989429a6e7d6b295033151bd88877c0dc82857e7277"} Aug 04 16:07:46.680166 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.680135 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs"] Aug 04 16:07:46.685444 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.685425 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:07:46.689745 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.689727 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9s67t\"" Aug 04 16:07:46.698796 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.698774 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs"] Aug 04 16:07:46.739929 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.739904 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpz6r\" (UniqueName: \"kubernetes.io/projected/f47189c6-0bbc-493a-a99d-f8923b55f366-kube-api-access-fpz6r\") pod \"limitador-operator-controller-manager-6c549b6d85-mpvqs\" (UID: \"f47189c6-0bbc-493a-a99d-f8923b55f366\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:07:46.772787 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.772756 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:46.841079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.841042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpz6r\" (UniqueName: \"kubernetes.io/projected/f47189c6-0bbc-493a-a99d-f8923b55f366-kube-api-access-fpz6r\") pod \"limitador-operator-controller-manager-6c549b6d85-mpvqs\" (UID: \"f47189c6-0bbc-493a-a99d-f8923b55f366\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:07:46.853509 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.853478 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpz6r\" (UniqueName: \"kubernetes.io/projected/f47189c6-0bbc-493a-a99d-f8923b55f366-kube-api-access-fpz6r\") pod \"limitador-operator-controller-manager-6c549b6d85-mpvqs\" (UID: \"f47189c6-0bbc-493a-a99d-f8923b55f366\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:07:46.996736 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:46.996646 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:07:47.144570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:47.142499 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs"] Aug 04 16:07:47.344905 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:47.344864 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" event={"ID":"f47189c6-0bbc-493a-a99d-f8923b55f366","Type":"ContainerStarted","Data":"37a1d4f143446fa5ccf67ca3f1f5275b9f6ea13c37dca7a1ee98c495b8d66027"} Aug 04 16:07:47.408078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:47.408044 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:52.116592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:52.116558 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:52.340967 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:52.340940 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:52.376751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:52.376666 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" event={"ID":"f47189c6-0bbc-493a-a99d-f8923b55f366","Type":"ContainerStarted","Data":"f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831"} Aug 04 16:07:52.376751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:52.376747 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:07:52.453947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:52.453918 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:52.552195 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:52.552163 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:52.656861 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:52.656778 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:07:56.397619 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:56.397583 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" event={"ID":"60059be3-f43e-4806-a420-c34245f4fc23","Type":"ContainerStarted","Data":"c77662ef4a497b02c2aca2ee8b7e15086f143301ec6d869c80d8d82114071a01"} Aug 04 16:07:56.398001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:56.397752 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" Aug 04 16:07:56.417103 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:56.417056 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" podStartSLOduration=2.43102566 podStartE2EDuration="13.417040254s" totalImagesPullingTime="10.986014594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:07:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:07:56.415390536 +0000 UTC m=+685.508990783" watchObservedRunningTime="2026-08-04 16:07:56.417040254 +0000 UTC m=+685.510640501" Aug 04 16:07:56.417814 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:07:56.417788 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" podStartSLOduration=5.812834575 podStartE2EDuration="10.417781822s" totalImagesPullingTime="4.604947247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:07:52.414041352 +0000 UTC m=+681.507641599" watchObservedRunningTime="2026-08-04 16:07:56.417781822 +0000 UTC m=+685.511382073" Aug 04 16:08:00.510925 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:00.510894 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:00.610647 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:00.610603 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:03.107224 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.107175 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs"] Aug 04 16:08:03.107677 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.107408 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" containerName="manager" containerID="cri-o://f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831" gracePeriod=2 Aug 04 16:08:03.109684 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.109653 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:08:03.125699 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.125588 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs"] Aug 04 16:08:03.135607 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:08:03.135573 2571 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f47189c6-0bbc-493a-a99d-f8923b55f366\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-mpvqs\": pods \"limitador-operator-controller-manager-6c549b6d85-mpvqs\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:08:03.141870 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:08:03.141838 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-mpvqs\" is forbidden: User \"system:node:ip-10-0-137-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-14.ec2.internal' and this object" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:08:03.367705 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.367648 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:08:03.436787 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.436753 2571 generic.go:356] "Generic (PLEG): container finished" podID="f47189c6-0bbc-493a-a99d-f8923b55f366" containerID="f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831" exitCode=0 Aug 04 16:08:03.436947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.436797 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:08:03.436947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.436834 2571 scope.go:117] "RemoveContainer" containerID="f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831" Aug 04 16:08:03.445353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.445332 2571 scope.go:117] "RemoveContainer" containerID="f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831" Aug 04 16:08:03.445612 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:08:03.445591 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831\": container with ID starting with f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831 not found: ID does not exist" containerID="f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831" Aug 04 16:08:03.445659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.445621 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831"} err="failed to get container status \"f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831\": rpc error: code = NotFound desc = could not find container \"f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831\": container with ID starting with f13371b4f10bf1c8838c9f9592cba8574c745213027c5900131ec298954b0831 not found: ID does not exist" Aug 04 16:08:03.494011 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.493984 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpz6r\" (UniqueName: \"kubernetes.io/projected/f47189c6-0bbc-493a-a99d-f8923b55f366-kube-api-access-fpz6r\") pod \"f47189c6-0bbc-493a-a99d-f8923b55f366\" (UID: \"f47189c6-0bbc-493a-a99d-f8923b55f366\") " Aug 04 16:08:03.496020 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.495995 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47189c6-0bbc-493a-a99d-f8923b55f366-kube-api-access-fpz6r" (OuterVolumeSpecName: "kube-api-access-fpz6r") pod "f47189c6-0bbc-493a-a99d-f8923b55f366" (UID: "f47189c6-0bbc-493a-a99d-f8923b55f366"). InnerVolumeSpecName "kube-api-access-fpz6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:08:03.553002 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.552976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:03.595085 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.595054 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpz6r\" (UniqueName: \"kubernetes.io/projected/f47189c6-0bbc-493a-a99d-f8923b55f366-kube-api-access-fpz6r\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:08:03.660870 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:03.660791 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:03.754560 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:08:03.754529 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-mpvqs\" is forbidden: User \"system:node:ip-10-0-137-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-14.ec2.internal' and this object" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:08:04.966787 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:04.966754 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9"] Aug 04 16:08:04.967154 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:04.967108 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" containerName="manager" Aug 04 16:08:04.967154 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:04.967120 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" containerName="manager" Aug 04 16:08:04.967258 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:04.967173 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" containerName="manager" Aug 04 16:08:04.971353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:04.971336 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:04.975495 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:04.975472 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vhhgd\"" Aug 04 16:08:04.991505 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:04.991483 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9"] Aug 04 16:08:05.016215 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:08:05.016174 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-mpvqs\" is forbidden: User \"system:node:ip-10-0-137-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-14.ec2.internal' and this object" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-mpvqs" Aug 04 16:08:05.108690 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.108661 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/692c82ab-a892-4f2b-8b50-36c469983fb0-kube-api-access-h5zm7\") pod \"kuadrant-operator-controller-manager-68cf589589-br8j9\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:05.108822 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.108697 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/692c82ab-a892-4f2b-8b50-36c469983fb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-br8j9\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:05.209869 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.209836 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/692c82ab-a892-4f2b-8b50-36c469983fb0-kube-api-access-h5zm7\") pod \"kuadrant-operator-controller-manager-68cf589589-br8j9\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:05.210010 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.209877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/692c82ab-a892-4f2b-8b50-36c469983fb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-br8j9\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:05.210276 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.210259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/692c82ab-a892-4f2b-8b50-36c469983fb0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-br8j9\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:05.221065 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.221010 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/692c82ab-a892-4f2b-8b50-36c469983fb0-kube-api-access-h5zm7\") pod \"kuadrant-operator-controller-manager-68cf589589-br8j9\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:05.280759 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.280737 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:05.363513 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.363023 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f47189c6-0bbc-493a-a99d-f8923b55f366" path="/var/lib/kubelet/pods/f47189c6-0bbc-493a-a99d-f8923b55f366/volumes" Aug 04 16:08:05.428892 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.428870 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9"] Aug 04 16:08:05.430255 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:08:05.430227 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692c82ab_a892_4f2b_8b50_36c469983fb0.slice/crio-87516c9bd394d36692b4fea0716ccba7cdf9893dd6ceb08f2a54d48745f32f25 WatchSource:0}: Error finding container 87516c9bd394d36692b4fea0716ccba7cdf9893dd6ceb08f2a54d48745f32f25: Status 404 returned error can't find the container with id 87516c9bd394d36692b4fea0716ccba7cdf9893dd6ceb08f2a54d48745f32f25 Aug 04 16:08:05.449294 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:05.449268 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" event={"ID":"692c82ab-a892-4f2b-8b50-36c469983fb0","Type":"ContainerStarted","Data":"87516c9bd394d36692b4fea0716ccba7cdf9893dd6ceb08f2a54d48745f32f25"} Aug 04 16:08:06.056317 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:06.056279 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:06.152953 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:06.152918 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:07.405356 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:07.405324 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-c8jf6" Aug 04 16:08:07.888686 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:07.888631 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:08.004417 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:08.004384 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:09.481533 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:09.481488 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" event={"ID":"692c82ab-a892-4f2b-8b50-36c469983fb0","Type":"ContainerStarted","Data":"9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49"} Aug 04 16:08:09.481931 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:09.481590 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:09.517765 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:09.517722 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" podStartSLOduration=1.946826799 podStartE2EDuration="5.517706868s" totalImagesPullingTime="3.570880069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:08:09.51675451 +0000 UTC m=+698.610354763" watchObservedRunningTime="2026-08-04 16:08:09.517706868 +0000 UTC m=+698.611307116" Aug 04 16:08:13.150907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:13.150874 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:13.290258 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:13.290217 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:13.418546 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:13.418466 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:13.547743 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:13.547710 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:13.652805 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:13.652772 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:20.490087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:20.490057 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:08:21.960055 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:21.960024 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:22.068822 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:22.068788 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:24.996770 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:24.996688 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:25.096183 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:25.096150 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:27.382397 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:27.382365 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:27.449057 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:27.449026 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:33.842339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:33.842301 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:33.945530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:33.945496 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:36.951448 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:36.951411 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:37.499437 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:37.499405 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:39.606773 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:39.606736 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:39.907098 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:39.907020 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:08:39.910394 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:39.910373 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:39.913195 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:39.913177 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 16:08:39.913305 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:39.913257 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vk9xc\"" Aug 04 16:08:39.925936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:39.925909 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:08:39.963002 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:39.962969 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:08:40.007130 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.007100 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a94ae9e4-8d3d-45fc-96ec-648c2690c45e-config-file\") pod \"limitador-limitador-85d9db4d5d-5h7sv\" (UID: \"a94ae9e4-8d3d-45fc-96ec-648c2690c45e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:40.007304 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.007141 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvd2\" (UniqueName: \"kubernetes.io/projected/a94ae9e4-8d3d-45fc-96ec-648c2690c45e-kube-api-access-vjvd2\") pod \"limitador-limitador-85d9db4d5d-5h7sv\" (UID: \"a94ae9e4-8d3d-45fc-96ec-648c2690c45e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:40.108517 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.108484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a94ae9e4-8d3d-45fc-96ec-648c2690c45e-config-file\") pod \"limitador-limitador-85d9db4d5d-5h7sv\" (UID: \"a94ae9e4-8d3d-45fc-96ec-648c2690c45e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:40.108678 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.108536 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvd2\" (UniqueName: \"kubernetes.io/projected/a94ae9e4-8d3d-45fc-96ec-648c2690c45e-kube-api-access-vjvd2\") pod \"limitador-limitador-85d9db4d5d-5h7sv\" (UID: \"a94ae9e4-8d3d-45fc-96ec-648c2690c45e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:40.109111 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.109091 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a94ae9e4-8d3d-45fc-96ec-648c2690c45e-config-file\") pod \"limitador-limitador-85d9db4d5d-5h7sv\" (UID: \"a94ae9e4-8d3d-45fc-96ec-648c2690c45e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:40.121008 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.120972 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvd2\" (UniqueName: \"kubernetes.io/projected/a94ae9e4-8d3d-45fc-96ec-648c2690c45e-kube-api-access-vjvd2\") pod \"limitador-limitador-85d9db4d5d-5h7sv\" (UID: \"a94ae9e4-8d3d-45fc-96ec-648c2690c45e\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:40.221173 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.221095 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:40.365780 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.365747 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:08:40.367698 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:08:40.367667 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94ae9e4_8d3d_45fc_96ec_648c2690c45e.slice/crio-9df9bcce5157cbe3132a856fdffc86cdc557ac9cd60c0ef201e8fb4f996e178c WatchSource:0}: Error finding container 9df9bcce5157cbe3132a856fdffc86cdc557ac9cd60c0ef201e8fb4f996e178c: Status 404 returned error can't find the container with id 9df9bcce5157cbe3132a856fdffc86cdc557ac9cd60c0ef201e8fb4f996e178c Aug 04 16:08:40.443073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.443043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:40.535597 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.535516 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:40.652594 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:40.652560 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" event={"ID":"a94ae9e4-8d3d-45fc-96ec-648c2690c45e","Type":"ContainerStarted","Data":"9df9bcce5157cbe3132a856fdffc86cdc557ac9cd60c0ef201e8fb4f996e178c"} Aug 04 16:08:41.659691 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:41.659658 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" event={"ID":"a94ae9e4-8d3d-45fc-96ec-648c2690c45e","Type":"ContainerStarted","Data":"733d24a80c72543861bab02ca129acc18faed5cd125a5708fadfc12116f45b92"} Aug 04 16:08:41.660073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:41.659723 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:41.682624 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:41.682579 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" podStartSLOduration=1.4953691660000001 podStartE2EDuration="2.682563821s" totalImagesPullingTime="1.187194655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:08:41.679267281 +0000 UTC m=+730.772867534" watchObservedRunningTime="2026-08-04 16:08:41.682563821 +0000 UTC m=+730.776164075" Aug 04 16:08:52.666152 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:52.666123 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv" Aug 04 16:08:54.581551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:54.581515 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:55.559512 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:55.559480 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:55.657539 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:55.657505 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:08:56.346856 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:08:56.346827 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:07.849016 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:07.848980 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:08.004007 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:08.003976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:16.517123 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:16.517089 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:16.620017 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:16.619982 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:17.119105 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:17.119073 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:19.950583 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:19.950553 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:20.126843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:20.126811 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:27.795585 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:27.795547 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:52.833491 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:52.833454 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:52.952308 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:52.952272 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:54.555751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:54.555718 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:54.872663 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:54.872579 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:55.118722 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:55.118689 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:55.228522 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:55.228442 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:55.585198 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:09:55.585165 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:10:05.273723 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:05.273689 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:05.392357 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:05.392320 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:13.579815 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:13.579773 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:13.907249 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:13.907157 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:10:23.514847 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:23.514816 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:24.089681 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:24.089647 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:10:35.053914 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:35.053879 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:35.989184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:35.989146 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:10:43.505397 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:43.505364 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:44.593096 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:44.593060 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:10:55.827144 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.827108 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw"] Aug 04 16:10:55.830783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.830766 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:55.834424 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.834401 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 16:10:55.835707 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.835686 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zcdht\"" Aug 04 16:10:55.835707 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.835699 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Aug 04 16:10:55.835861 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.835761 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 16:10:55.850115 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.850087 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw"] Aug 04 16:10:55.911514 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:55.911478 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:56.012977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.012946 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.013147 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.012986 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.013147 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.013045 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.013147 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.013121 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87e1f70-ca9d-4586-8b89-fe132171c3f7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.013273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.013159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.013273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.013250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb9x\" (UniqueName: \"kubernetes.io/projected/c87e1f70-ca9d-4586-8b89-fe132171c3f7-kube-api-access-ddb9x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.113911 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb9x\" (UniqueName: \"kubernetes.io/projected/c87e1f70-ca9d-4586-8b89-fe132171c3f7-kube-api-access-ddb9x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.113955 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.113976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.114014 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.114056 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87e1f70-ca9d-4586-8b89-fe132171c3f7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.114090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114494 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.114472 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.114491 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.114551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.114531 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.116217 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.116184 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87e1f70-ca9d-4586-8b89-fe132171c3f7-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.116510 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.116493 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87e1f70-ca9d-4586-8b89-fe132171c3f7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.123113 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.123085 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb9x\" (UniqueName: \"kubernetes.io/projected/c87e1f70-ca9d-4586-8b89-fe132171c3f7-kube-api-access-ddb9x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-rsxkw\" (UID: \"c87e1f70-ca9d-4586-8b89-fe132171c3f7\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.140992 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.140969 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:10:56.286034 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.286006 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw"] Aug 04 16:10:56.287421 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:10:56.287394 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87e1f70_ca9d_4586_8b89_fe132171c3f7.slice/crio-86a72ec13f663a352b305ea9cc482c9a6bfc1d46d6e729f09ea154e6bf27d755 WatchSource:0}: Error finding container 86a72ec13f663a352b305ea9cc482c9a6bfc1d46d6e729f09ea154e6bf27d755: Status 404 returned error can't find the container with id 86a72ec13f663a352b305ea9cc482c9a6bfc1d46d6e729f09ea154e6bf27d755 Aug 04 16:10:56.288651 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.288633 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:10:56.402774 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:56.402684 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" event={"ID":"c87e1f70-ca9d-4586-8b89-fe132171c3f7","Type":"ContainerStarted","Data":"86a72ec13f663a352b305ea9cc482c9a6bfc1d46d6e729f09ea154e6bf27d755"} Aug 04 16:10:58.132126 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:58.132091 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:10:58.416125 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:58.416034 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" event={"ID":"c87e1f70-ca9d-4586-8b89-fe132171c3f7","Type":"ContainerStarted","Data":"f1cf79e94586eb38becc369a88e256f800673527f0bf98445796683e92ed45d0"} Aug 04 16:10:58.436744 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:10:58.436693 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" podStartSLOduration=2.338686949 podStartE2EDuration="3.436677572s" totalImagesPullingTime="1.097990623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:10:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:10:58.436061194 +0000 UTC m=+867.529661441" watchObservedRunningTime="2026-08-04 16:10:58.436677572 +0000 UTC m=+867.530277819" Aug 04 16:11:06.141140 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:11:06.141091 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:11:06.141700 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:11:06.141171 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:11:06.153672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:11:06.153646 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:11:06.472507 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:11:06.472423 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-rsxkw" Aug 04 16:11:13.225638 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:11:13.225603 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:15.588993 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:11:15.588959 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:11:39.504979 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:11:39.504937 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:08.202302 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:08.202268 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:08.293795 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:08.293763 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:12.309568 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:12.309532 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:12.443803 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:12.443755 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:38.695243 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:38.695216 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:12:40.605968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:40.605933 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:47.388653 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:47.388604 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:12:59.190441 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:59.190365 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:12:59.875000 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:12:59.874966 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:39.624399 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:13:39.624345 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:39.723294 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:13:39.723267 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:15.572030 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:14:15.571997 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:17.181117 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:14:17.181087 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:38.101712 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:14:38.101626 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:14:45.873447 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:14:45.873396 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:48.108603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:14:48.108572 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:15:17.287088 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:17.287043 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:17.376999 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:17.376967 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:17.541356 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:17.541266 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:17.894371 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:17.894324 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:23.614699 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:23.614659 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:23.728792 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:23.728758 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:23.934923 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:23.934844 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:24.072100 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:24.072066 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:30.761386 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:30.761340 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:30.905728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:30.905692 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:46.696275 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:46.696239 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:15:47.364240 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:47.364191 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:47.430322 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:47.430288 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:15:53.787265 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:15:53.787155 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:16:09.367031 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:16:09.367002 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:16:15.095871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:16:15.095838 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:16:24.898825 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:16:24.898788 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:16:31.590660 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:16:31.590620 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:16:45.690391 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:16:45.690352 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:16:50.788106 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:16:50.788058 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:16:55.981735 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:16:55.981688 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:17:02.714485 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:17:02.714446 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:17:08.289818 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:17:08.289778 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:17:21.295573 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:17:21.295528 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:17:32.118541 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:17:32.118454 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:17:56.593985 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:17:56.593950 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:18:07.197449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:18:07.197395 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:18:13.892619 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:18:13.892583 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:18:22.181549 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:18:22.181510 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:18:33.086969 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:18:33.086931 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:18:38.998062 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:18:38.998024 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:18:47.684155 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:18:47.684120 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:18:55.483680 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:18:55.483603 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:19:07.785838 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:19:07.785804 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:19:17.287561 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:19:17.287528 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:19:25.087080 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:19:25.087044 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:19:32.783164 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:19:32.783127 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:19:37.989692 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:19:37.989645 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:19:46.888387 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:19:46.888351 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:19:53.207516 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:19:53.207480 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:20:49.234423 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.234385 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9"] Aug 04 16:20:49.234980 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.234617 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" podUID="692c82ab-a892-4f2b-8b50-36c469983fb0" containerName="manager" containerID="cri-o://9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49" gracePeriod=10 Aug 04 16:20:49.478174 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.478150 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:20:49.618493 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.618458 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/692c82ab-a892-4f2b-8b50-36c469983fb0-kube-api-access-h5zm7\") pod \"692c82ab-a892-4f2b-8b50-36c469983fb0\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " Aug 04 16:20:49.618663 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.618532 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/692c82ab-a892-4f2b-8b50-36c469983fb0-extensions-socket-volume\") pod \"692c82ab-a892-4f2b-8b50-36c469983fb0\" (UID: \"692c82ab-a892-4f2b-8b50-36c469983fb0\") " Aug 04 16:20:49.618943 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.618919 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692c82ab-a892-4f2b-8b50-36c469983fb0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "692c82ab-a892-4f2b-8b50-36c469983fb0" (UID: "692c82ab-a892-4f2b-8b50-36c469983fb0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:20:49.620457 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.620436 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692c82ab-a892-4f2b-8b50-36c469983fb0-kube-api-access-h5zm7" (OuterVolumeSpecName: "kube-api-access-h5zm7") pod "692c82ab-a892-4f2b-8b50-36c469983fb0" (UID: "692c82ab-a892-4f2b-8b50-36c469983fb0"). InnerVolumeSpecName "kube-api-access-h5zm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:20:49.665667 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.665626 2571 generic.go:356] "Generic (PLEG): container finished" podID="692c82ab-a892-4f2b-8b50-36c469983fb0" containerID="9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49" exitCode=0 Aug 04 16:20:49.665667 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.665657 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" event={"ID":"692c82ab-a892-4f2b-8b50-36c469983fb0","Type":"ContainerDied","Data":"9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49"} Aug 04 16:20:49.665852 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.665718 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" event={"ID":"692c82ab-a892-4f2b-8b50-36c469983fb0","Type":"ContainerDied","Data":"87516c9bd394d36692b4fea0716ccba7cdf9893dd6ceb08f2a54d48745f32f25"} Aug 04 16:20:49.665852 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.665742 2571 scope.go:117] "RemoveContainer" containerID="9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49" Aug 04 16:20:49.665852 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.665667 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9" Aug 04 16:20:49.675978 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.675959 2571 scope.go:117] "RemoveContainer" containerID="9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49" Aug 04 16:20:49.676237 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:20:49.676216 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49\": container with ID starting with 9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49 not found: ID does not exist" containerID="9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49" Aug 04 16:20:49.676298 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.676246 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49"} err="failed to get container status \"9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49\": rpc error: code = NotFound desc = could not find container \"9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49\": container with ID starting with 9e590c3d2e2eb78c027181f241cfae2691243a13c51105191961472ebaea9f49 not found: ID does not exist" Aug 04 16:20:49.689889 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.689865 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9"] Aug 04 16:20:49.693821 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.693799 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-br8j9"] Aug 04 16:20:49.719772 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.719755 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zm7\" (UniqueName: \"kubernetes.io/projected/692c82ab-a892-4f2b-8b50-36c469983fb0-kube-api-access-h5zm7\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:20:49.719772 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:49.719773 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/692c82ab-a892-4f2b-8b50-36c469983fb0-extensions-socket-volume\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:20:51.364285 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:20:51.364246 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692c82ab-a892-4f2b-8b50-36c469983fb0" path="/var/lib/kubelet/pods/692c82ab-a892-4f2b-8b50-36c469983fb0/volumes" Aug 04 16:21:16.932129 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:16.932096 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:21:19.397950 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:19.397917 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:21:29.144296 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:29.144256 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:21:30.449544 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:30.449510 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:21:49.999584 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:49.999555 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq"] Aug 04 16:21:50.001857 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:49.999923 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692c82ab-a892-4f2b-8b50-36c469983fb0" containerName="manager" Aug 04 16:21:50.001857 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:49.999934 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="692c82ab-a892-4f2b-8b50-36c469983fb0" containerName="manager" Aug 04 16:21:50.001857 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.000008 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="692c82ab-a892-4f2b-8b50-36c469983fb0" containerName="manager" Aug 04 16:21:50.002798 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.002780 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.005710 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.005693 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vhhgd\"" Aug 04 16:21:50.014544 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.014524 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq"] Aug 04 16:21:50.138537 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.138506 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/50eddab3-9851-49f9-9121-c184c7435c26-kube-api-access-s4nfg\") pod \"kuadrant-operator-controller-manager-68cf589589-xd8hq\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.138727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.138555 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50eddab3-9851-49f9-9121-c184c7435c26-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-xd8hq\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.239140 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.239105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/50eddab3-9851-49f9-9121-c184c7435c26-kube-api-access-s4nfg\") pod \"kuadrant-operator-controller-manager-68cf589589-xd8hq\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.239305 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.239148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50eddab3-9851-49f9-9121-c184c7435c26-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-xd8hq\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.239485 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.239468 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50eddab3-9851-49f9-9121-c184c7435c26-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-xd8hq\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.248569 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.248550 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/50eddab3-9851-49f9-9121-c184c7435c26-kube-api-access-s4nfg\") pod \"kuadrant-operator-controller-manager-68cf589589-xd8hq\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.313627 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.313603 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:50.468142 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:50.468051 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq"] Aug 04 16:21:50.470751 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:21:50.470722 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eddab3_9851_49f9_9121_c184c7435c26.slice/crio-a7c9552c9f2727842cf7a86b5e5821e61f695c09f0a7f4b8012584af94a81b72 WatchSource:0}: Error finding container a7c9552c9f2727842cf7a86b5e5821e61f695c09f0a7f4b8012584af94a81b72: Status 404 returned error can't find the container with id a7c9552c9f2727842cf7a86b5e5821e61f695c09f0a7f4b8012584af94a81b72 Aug 04 16:21:51.003375 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:51.003340 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" event={"ID":"50eddab3-9851-49f9-9121-c184c7435c26","Type":"ContainerStarted","Data":"199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe"} Aug 04 16:21:51.003771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:51.003394 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" event={"ID":"50eddab3-9851-49f9-9121-c184c7435c26","Type":"ContainerStarted","Data":"a7c9552c9f2727842cf7a86b5e5821e61f695c09f0a7f4b8012584af94a81b72"} Aug 04 16:21:51.003771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:51.003419 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:21:51.028695 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:21:51.028648 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" podStartSLOduration=2.028632744 podStartE2EDuration="2.028632744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:21:51.026009978 +0000 UTC m=+1520.119610226" watchObservedRunningTime="2026-08-04 16:21:51.028632744 +0000 UTC m=+1520.122232992" Aug 04 16:22:02.010080 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:22:02.010049 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:22:11.494293 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:22:11.494244 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:22:16.480353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:22:16.480312 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:22:42.397663 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:22:42.397625 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:23:12.218646 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:23:12.218612 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:23:12.307405 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:23:12.307371 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:23:20.984887 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:23:20.984853 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:23:27.584820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:23:27.584740 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:23:36.990359 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:23:36.990323 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:23:42.598227 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:23:42.598169 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:23:53.592121 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:23:53.592086 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:24:03.090194 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:24:03.090154 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:24:08.708447 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:24:08.708413 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:24:13.689557 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:24:13.689519 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:24:23.091930 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:24:23.091892 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:24:29.788915 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:24:29.788876 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:24:38.986653 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:24:38.986612 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:25:03.551401 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:25:03.551324 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:25:08.600803 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:25:08.600763 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:25:11.059837 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:25:11.059803 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:25:15.085700 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:25:15.085659 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:25:36.929990 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:25:36.929955 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:25:39.394226 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:25:39.394164 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:26:13.043094 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:13.043053 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:26:14.190648 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:14.190592 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:26:15.383449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:15.383413 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:26:26.787881 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:26.787801 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:26:34.298654 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:34.298615 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:26:42.184443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:42.184406 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:26:49.686402 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:49.686367 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:26:57.083612 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:26:57.083571 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:27:03.277854 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:27:03.277820 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:27:08.278368 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:27:08.278329 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:27:17.290964 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:27:17.290928 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:27:24.786717 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:27:24.786683 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:27:40.184759 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:27:40.184719 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:27:47.490391 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:27:47.490355 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:27:58.783515 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:27:58.783430 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:06.382233 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:06.381059 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:12.598561 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:12.598524 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:18.284935 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:18.284898 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:24.589525 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:24.589487 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:34.190874 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:34.190838 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:41.187863 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:41.187823 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:45.802476 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:45.802442 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:28:51.278604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:51.278565 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:28:58.384596 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:28:58.384562 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:29:08.581459 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:08.581418 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:29:16.907862 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:16.907822 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:29:24.097747 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:24.097649 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:29:30.976874 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:30.976839 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:29:33.523600 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:33.523560 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:29:41.481295 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:41.481257 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:29:48.887078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:48.887038 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:29:56.991678 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:56.991641 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:29:59.578773 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:29:59.578733 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:02.699148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:02.699098 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:02.786231 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:02.783310 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:03.227138 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:03.227103 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:05.785141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:05.785100 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:30:09.206119 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:09.206075 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:09.350518 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:09.350485 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:09.463089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:09.462994 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:11.147255 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:11.147216 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:11.278463 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:11.278428 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:12.926020 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:12.925976 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:12.986508 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:12.986466 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:13.465266 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:13.465225 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:14.083664 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:14.083635 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:14.182944 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:14.182903 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:15.979681 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:15.979646 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:30:37.845797 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:37.845756 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8"] Aug 04 16:30:37.849129 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:37.849111 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" Aug 04 16:30:37.851992 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:37.851959 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:30:37.851992 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:37.851977 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:30:37.858915 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:37.858884 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8"] Aug 04 16:30:37.994831 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:37.994783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2baef5-0a68-4975-8a91-4f42343f6419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8\" (UID: \"ba2baef5-0a68-4975-8a91-4f42343f6419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" Aug 04 16:30:38.096019 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:38.095921 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2baef5-0a68-4975-8a91-4f42343f6419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8\" (UID: \"ba2baef5-0a68-4975-8a91-4f42343f6419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" Aug 04 16:30:38.096583 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:38.096560 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2baef5-0a68-4975-8a91-4f42343f6419-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8\" (UID: \"ba2baef5-0a68-4975-8a91-4f42343f6419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" Aug 04 16:30:38.161021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:38.160978 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" Aug 04 16:30:38.300763 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:38.300729 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8"] Aug 04 16:30:38.303999 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:30:38.303967 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2baef5_0a68_4975_8a91_4f42343f6419.slice/crio-81862569d5602024f2839fd95877c76190fe04539e3be0da603351bfcfe0b698 WatchSource:0}: Error finding container 81862569d5602024f2839fd95877c76190fe04539e3be0da603351bfcfe0b698: Status 404 returned error can't find the container with id 81862569d5602024f2839fd95877c76190fe04539e3be0da603351bfcfe0b698 Aug 04 16:30:38.305960 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:38.305944 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:30:38.930060 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:38.930025 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" event={"ID":"ba2baef5-0a68-4975-8a91-4f42343f6419","Type":"ContainerStarted","Data":"81862569d5602024f2839fd95877c76190fe04539e3be0da603351bfcfe0b698"} Aug 04 16:30:41.949004 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:41.948964 2571 generic.go:356] "Generic (PLEG): container finished" podID="ba2baef5-0a68-4975-8a91-4f42343f6419" containerID="28b56d687f38e93f4bde83de1af3fa7ef6c5900076245d750bdfd642b1bfac5a" exitCode=0 Aug 04 16:30:41.949547 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:41.949022 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" event={"ID":"ba2baef5-0a68-4975-8a91-4f42343f6419","Type":"ContainerDied","Data":"28b56d687f38e93f4bde83de1af3fa7ef6c5900076245d750bdfd642b1bfac5a"} Aug 04 16:30:43.085765 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:43.085738 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" Aug 04 16:30:43.241730 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:43.241643 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2baef5-0a68-4975-8a91-4f42343f6419-maas-api-service-ca\") pod \"ba2baef5-0a68-4975-8a91-4f42343f6419\" (UID: \"ba2baef5-0a68-4975-8a91-4f42343f6419\") " Aug 04 16:30:43.242040 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:43.242014 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2baef5-0a68-4975-8a91-4f42343f6419-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba2baef5-0a68-4975-8a91-4f42343f6419" (UID: "ba2baef5-0a68-4975-8a91-4f42343f6419"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:30:43.342285 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:43.342243 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba2baef5-0a68-4975-8a91-4f42343f6419-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:30:43.961427 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:43.961398 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" Aug 04 16:30:43.961427 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:43.961418 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8" event={"ID":"ba2baef5-0a68-4975-8a91-4f42343f6419","Type":"ContainerDied","Data":"81862569d5602024f2839fd95877c76190fe04539e3be0da603351bfcfe0b698"} Aug 04 16:30:43.961645 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:43.961464 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81862569d5602024f2839fd95877c76190fe04539e3be0da603351bfcfe0b698" Aug 04 16:30:46.620038 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:46.620000 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:48.033863 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:48.033829 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:53.973116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:53.973016 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:54.073418 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.073376 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:54.162939 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.162906 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8"] Aug 04 16:30:54.170180 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.170148 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a345e2e-gxkc8"] Aug 04 16:30:54.170412 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.170385 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:54.322973 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.322936 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:54.555315 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.555278 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:54.839096 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.839059 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:54.941894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:54.941857 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:55.039114 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:55.039078 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:55.363627 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:55.363590 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba2baef5-0a68-4975-8a91-4f42343f6419" path="/var/lib/kubelet/pods/ba2baef5-0a68-4975-8a91-4f42343f6419/volumes" Aug 04 16:30:56.813612 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:56.813575 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:56.868112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:56.868073 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:57.094247 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:57.094150 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:30:57.679376 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:57.679343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:57.781197 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:57.781160 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:57.984341 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:57.984233 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:58.423265 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:58.423189 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:30:58.521027 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:30:58.520977 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:03.777273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:03.777229 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:31:22.714404 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.714369 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv"] Aug 04 16:31:22.714925 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.714753 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2baef5-0a68-4975-8a91-4f42343f6419" containerName="revoke-keys" Aug 04 16:31:22.714925 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.714765 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2baef5-0a68-4975-8a91-4f42343f6419" containerName="revoke-keys" Aug 04 16:31:22.714925 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.714834 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2baef5-0a68-4975-8a91-4f42343f6419" containerName="revoke-keys" Aug 04 16:31:22.720545 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.720525 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" Aug 04 16:31:22.723947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.723739 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:31:22.723947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.723776 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:31:22.726124 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.726091 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv"] Aug 04 16:31:22.819322 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.819288 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb56f910-5e2d-4404-8b45-e4d78fa8f682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-64639142-qbddv\" (UID: \"bb56f910-5e2d-4404-8b45-e4d78fa8f682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" Aug 04 16:31:22.920660 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.920622 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb56f910-5e2d-4404-8b45-e4d78fa8f682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-64639142-qbddv\" (UID: \"bb56f910-5e2d-4404-8b45-e4d78fa8f682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" Aug 04 16:31:22.921274 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:22.921251 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb56f910-5e2d-4404-8b45-e4d78fa8f682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-64639142-qbddv\" (UID: \"bb56f910-5e2d-4404-8b45-e4d78fa8f682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" Aug 04 16:31:23.033455 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:23.033372 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" Aug 04 16:31:23.174510 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:23.174469 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv"] Aug 04 16:31:24.194440 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:24.194344 2571 generic.go:356] "Generic (PLEG): container finished" podID="bb56f910-5e2d-4404-8b45-e4d78fa8f682" containerID="5026e15d3f1b8fd41f11d6edc5fbdef5df77e393be33f1b81561bc8aeb989aad" exitCode=0 Aug 04 16:31:24.194824 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:24.194430 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" event={"ID":"bb56f910-5e2d-4404-8b45-e4d78fa8f682","Type":"ContainerDied","Data":"5026e15d3f1b8fd41f11d6edc5fbdef5df77e393be33f1b81561bc8aeb989aad"} Aug 04 16:31:24.194824 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:24.194487 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" event={"ID":"bb56f910-5e2d-4404-8b45-e4d78fa8f682","Type":"ContainerStarted","Data":"1dca6c3c0cd1950c8b78b6825ca49ce9f94aa31a5a8fcb70f74516b59c84e514"} Aug 04 16:31:25.335461 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:25.335433 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" Aug 04 16:31:25.443291 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:25.443257 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb56f910-5e2d-4404-8b45-e4d78fa8f682-maas-api-service-ca\") pod \"bb56f910-5e2d-4404-8b45-e4d78fa8f682\" (UID: \"bb56f910-5e2d-4404-8b45-e4d78fa8f682\") " Aug 04 16:31:25.443603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:25.443577 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb56f910-5e2d-4404-8b45-e4d78fa8f682-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb56f910-5e2d-4404-8b45-e4d78fa8f682" (UID: "bb56f910-5e2d-4404-8b45-e4d78fa8f682"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:31:25.544832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:25.544743 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb56f910-5e2d-4404-8b45-e4d78fa8f682-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:31:26.207046 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:26.207015 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" Aug 04 16:31:26.207046 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:26.207027 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv" event={"ID":"bb56f910-5e2d-4404-8b45-e4d78fa8f682","Type":"ContainerDied","Data":"1dca6c3c0cd1950c8b78b6825ca49ce9f94aa31a5a8fcb70f74516b59c84e514"} Aug 04 16:31:26.207279 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:26.207069 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dca6c3c0cd1950c8b78b6825ca49ce9f94aa31a5a8fcb70f74516b59c84e514" Aug 04 16:31:29.212607 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.212568 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:29.278759 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.278724 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:29.348076 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.348033 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:29.387761 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.387727 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv"] Aug 04 16:31:29.390486 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.390460 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-64639142-qbddv"] Aug 04 16:31:29.407970 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.407941 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:29.562390 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.562358 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:29.840611 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:29.840519 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:30.089446 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:30.089409 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:30.459542 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:30.459505 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:30.707280 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:30.707244 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:31.363637 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:31.363607 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb56f910-5e2d-4404-8b45-e4d78fa8f682" path="/var/lib/kubelet/pods/bb56f910-5e2d-4404-8b45-e4d78fa8f682/volumes" Aug 04 16:31:32.088997 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:32.088962 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:31:32.745559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:32.745520 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:32.831410 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:32.831364 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:33.877389 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:33.877349 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:34.010079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:34.010045 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:34.375824 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:34.375793 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:34.484711 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:34.484675 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:31:40.484390 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:40.484359 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:31:58.617501 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.617464 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz"] Aug 04 16:31:58.619879 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.617846 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb56f910-5e2d-4404-8b45-e4d78fa8f682" containerName="revoke-keys" Aug 04 16:31:58.619879 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.617857 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb56f910-5e2d-4404-8b45-e4d78fa8f682" containerName="revoke-keys" Aug 04 16:31:58.619879 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.617935 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb56f910-5e2d-4404-8b45-e4d78fa8f682" containerName="revoke-keys" Aug 04 16:31:58.621090 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.621065 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" Aug 04 16:31:58.623851 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.623829 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:31:58.623961 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.623876 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:31:58.635921 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.635894 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz"] Aug 04 16:31:58.714550 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.714514 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a45031-a091-4e04-8ac3-0895efb90a05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1978158b-vhspz\" (UID: \"35a45031-a091-4e04-8ac3-0895efb90a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" Aug 04 16:31:58.815028 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.814982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a45031-a091-4e04-8ac3-0895efb90a05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1978158b-vhspz\" (UID: \"35a45031-a091-4e04-8ac3-0895efb90a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" Aug 04 16:31:58.815681 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.815659 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a45031-a091-4e04-8ac3-0895efb90a05-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1978158b-vhspz\" (UID: \"35a45031-a091-4e04-8ac3-0895efb90a05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" Aug 04 16:31:58.931480 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:58.931372 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" Aug 04 16:31:59.070470 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:59.070438 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz"] Aug 04 16:31:59.072262 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:31:59.072233 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a45031_a091_4e04_8ac3_0895efb90a05.slice/crio-a9cf755373826181ad4d88e036d62919da076c3fa0de5b87741521f01841f824 WatchSource:0}: Error finding container a9cf755373826181ad4d88e036d62919da076c3fa0de5b87741521f01841f824: Status 404 returned error can't find the container with id a9cf755373826181ad4d88e036d62919da076c3fa0de5b87741521f01841f824 Aug 04 16:31:59.393346 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:59.393312 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" event={"ID":"35a45031-a091-4e04-8ac3-0895efb90a05","Type":"ContainerStarted","Data":"856a6ea7a5e5225f85d095128ca148cd74d0ae2b7282ac0c05a845b01ae697cb"} Aug 04 16:31:59.393527 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:59.393362 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" event={"ID":"35a45031-a091-4e04-8ac3-0895efb90a05","Type":"ContainerStarted","Data":"a9cf755373826181ad4d88e036d62919da076c3fa0de5b87741521f01841f824"} Aug 04 16:31:59.409024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:31:59.408971 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" podStartSLOduration=1.408954968 podStartE2EDuration="1.408954968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:31:59.408389393 +0000 UTC m=+2128.501989640" watchObservedRunningTime="2026-08-04 16:31:59.408954968 +0000 UTC m=+2128.502555215" Aug 04 16:32:00.400967 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:00.400935 2571 generic.go:356] "Generic (PLEG): container finished" podID="35a45031-a091-4e04-8ac3-0895efb90a05" containerID="856a6ea7a5e5225f85d095128ca148cd74d0ae2b7282ac0c05a845b01ae697cb" exitCode=0 Aug 04 16:32:00.401386 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:00.401033 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" event={"ID":"35a45031-a091-4e04-8ac3-0895efb90a05","Type":"ContainerDied","Data":"856a6ea7a5e5225f85d095128ca148cd74d0ae2b7282ac0c05a845b01ae697cb"} Aug 04 16:32:01.543538 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:01.543508 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" Aug 04 16:32:01.641725 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:01.641695 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a45031-a091-4e04-8ac3-0895efb90a05-maas-api-service-ca\") pod \"35a45031-a091-4e04-8ac3-0895efb90a05\" (UID: \"35a45031-a091-4e04-8ac3-0895efb90a05\") " Aug 04 16:32:01.642130 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:01.642103 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a45031-a091-4e04-8ac3-0895efb90a05-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35a45031-a091-4e04-8ac3-0895efb90a05" (UID: "35a45031-a091-4e04-8ac3-0895efb90a05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:32:01.742943 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:01.742853 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35a45031-a091-4e04-8ac3-0895efb90a05-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:32:02.412687 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:02.412658 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" Aug 04 16:32:02.412687 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:02.412679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz" event={"ID":"35a45031-a091-4e04-8ac3-0895efb90a05","Type":"ContainerDied","Data":"a9cf755373826181ad4d88e036d62919da076c3fa0de5b87741521f01841f824"} Aug 04 16:32:02.412896 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:02.412724 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9cf755373826181ad4d88e036d62919da076c3fa0de5b87741521f01841f824" Aug 04 16:32:04.518232 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:04.518179 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:04.632997 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:04.632957 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:04.721144 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:04.721105 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:04.747849 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:04.747819 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz"] Aug 04 16:32:04.760474 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:04.760446 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1978158b-vhspz"] Aug 04 16:32:04.800043 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:04.799962 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:05.363888 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:05.363851 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a45031-a091-4e04-8ac3-0895efb90a05" path="/var/lib/kubelet/pods/35a45031-a091-4e04-8ac3-0895efb90a05/volumes" Aug 04 16:32:05.989156 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:05.989116 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:06.216146 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:06.216115 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:06.441586 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:06.441547 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:06.772363 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:06.772262 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:07.080119 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:07.080079 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:07.300148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:07.300113 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:32:08.935457 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:08.935417 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:09.034284 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:09.034240 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:10.180563 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:10.180527 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:10.286214 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:10.286176 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:10.685606 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:10.685503 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:10.779949 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:10.779914 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:13.480724 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:13.480688 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:32:33.892319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.892230 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556"] Aug 04 16:32:33.892815 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.892644 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a45031-a091-4e04-8ac3-0895efb90a05" containerName="revoke-keys" Aug 04 16:32:33.892815 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.892655 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a45031-a091-4e04-8ac3-0895efb90a05" containerName="revoke-keys" Aug 04 16:32:33.892815 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.892718 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a45031-a091-4e04-8ac3-0895efb90a05" containerName="revoke-keys" Aug 04 16:32:33.896053 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.896037 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" Aug 04 16:32:33.898840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.898818 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:32:33.898963 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.898857 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:32:33.904610 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.904585 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556"] Aug 04 16:32:33.938930 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:33.938898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a11f5-4620-4732-99b1-8c3e4aa8cee2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0eb193d3-k2556\" (UID: \"668a11f5-4620-4732-99b1-8c3e4aa8cee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" Aug 04 16:32:34.039687 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:34.039656 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a11f5-4620-4732-99b1-8c3e4aa8cee2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0eb193d3-k2556\" (UID: \"668a11f5-4620-4732-99b1-8c3e4aa8cee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" Aug 04 16:32:34.040338 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:34.040299 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a11f5-4620-4732-99b1-8c3e4aa8cee2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0eb193d3-k2556\" (UID: \"668a11f5-4620-4732-99b1-8c3e4aa8cee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" Aug 04 16:32:34.207862 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:34.207780 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" Aug 04 16:32:34.345898 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:34.345868 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556"] Aug 04 16:32:34.348637 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:32:34.348604 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668a11f5_4620_4732_99b1_8c3e4aa8cee2.slice/crio-474d849a6014f182eecba333b8d83fde7c7ea263d863f560f91d9de68f3a4e8d WatchSource:0}: Error finding container 474d849a6014f182eecba333b8d83fde7c7ea263d863f560f91d9de68f3a4e8d: Status 404 returned error can't find the container with id 474d849a6014f182eecba333b8d83fde7c7ea263d863f560f91d9de68f3a4e8d Aug 04 16:32:34.596172 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:34.596135 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" event={"ID":"668a11f5-4620-4732-99b1-8c3e4aa8cee2","Type":"ContainerStarted","Data":"474d849a6014f182eecba333b8d83fde7c7ea263d863f560f91d9de68f3a4e8d"} Aug 04 16:32:35.603213 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:35.603176 2571 generic.go:356] "Generic (PLEG): container finished" podID="668a11f5-4620-4732-99b1-8c3e4aa8cee2" containerID="348abde6f98be7ac2740bd9ebd1dc9c1d853e8218fb8138cd3f2aa7302190461" exitCode=0 Aug 04 16:32:35.603595 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:35.603277 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" event={"ID":"668a11f5-4620-4732-99b1-8c3e4aa8cee2","Type":"ContainerDied","Data":"348abde6f98be7ac2740bd9ebd1dc9c1d853e8218fb8138cd3f2aa7302190461"} Aug 04 16:32:36.738604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:36.738577 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" Aug 04 16:32:36.767192 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:36.767163 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a11f5-4620-4732-99b1-8c3e4aa8cee2-maas-api-service-ca\") pod \"668a11f5-4620-4732-99b1-8c3e4aa8cee2\" (UID: \"668a11f5-4620-4732-99b1-8c3e4aa8cee2\") " Aug 04 16:32:36.767507 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:36.767484 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/668a11f5-4620-4732-99b1-8c3e4aa8cee2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "668a11f5-4620-4732-99b1-8c3e4aa8cee2" (UID: "668a11f5-4620-4732-99b1-8c3e4aa8cee2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:32:36.868614 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:36.868526 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/668a11f5-4620-4732-99b1-8c3e4aa8cee2-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:32:37.615298 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:37.615268 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" Aug 04 16:32:37.615508 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:37.615267 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556" event={"ID":"668a11f5-4620-4732-99b1-8c3e4aa8cee2","Type":"ContainerDied","Data":"474d849a6014f182eecba333b8d83fde7c7ea263d863f560f91d9de68f3a4e8d"} Aug 04 16:32:37.615508 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:37.615391 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="474d849a6014f182eecba333b8d83fde7c7ea263d863f560f91d9de68f3a4e8d" Aug 04 16:32:40.610479 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:40.610439 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:40.686657 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:40.686628 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:40.762510 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:40.762474 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:40.808942 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:40.808908 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556"] Aug 04 16:32:40.812223 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:40.812187 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0eb193d3-k2556"] Aug 04 16:32:40.827250 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:40.827220 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:40.975217 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:40.975122 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:41.208094 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:41.208057 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:41.364820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:41.364784 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="668a11f5-4620-4732-99b1-8c3e4aa8cee2" path="/var/lib/kubelet/pods/668a11f5-4620-4732-99b1-8c3e4aa8cee2/volumes" Aug 04 16:32:41.695080 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:41.694993 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:41.826341 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:41.826306 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:41.981416 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:41.981325 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:43.686408 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:43.686370 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:32:53.731028 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:53.730991 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:32:56.482791 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:32:56.482759 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:33:00.531929 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:00.531885 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:00.636137 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:00.636099 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:00.764636 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:00.764592 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:00.931844 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:00.931806 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:01.377233 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:01.377169 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:14.652138 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:14.652102 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:27.388460 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:27.388428 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:33.400467 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:33.400425 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:33.501658 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:33.501623 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:33.600524 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:33.600489 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:37.471914 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:37.471871 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:49.596811 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:49.596778 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:55.444905 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:55.444817 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:55.550461 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:55.550419 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:33:55.651873 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:33:55.651841 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:06.281382 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:06.281343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:08.763178 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.763140 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bc8d79ddd-l9hm4"] Aug 04 16:34:08.763564 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.763537 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668a11f5-4620-4732-99b1-8c3e4aa8cee2" containerName="revoke-keys" Aug 04 16:34:08.763564 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.763550 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="668a11f5-4620-4732-99b1-8c3e4aa8cee2" containerName="revoke-keys" Aug 04 16:34:08.763647 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.763638 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="668a11f5-4620-4732-99b1-8c3e4aa8cee2" containerName="revoke-keys" Aug 04 16:34:08.766552 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.766532 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.792159 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.792125 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc8d79ddd-l9hm4"] Aug 04 16:34:08.798199 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.798175 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-console-config\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.798317 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.798222 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvdz\" (UniqueName: \"kubernetes.io/projected/3d79a012-9886-4c90-8286-f885ae0adcab-kube-api-access-8gvdz\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.798317 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.798298 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-oauth-serving-cert\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.798401 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.798352 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-trusted-ca-bundle\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.798437 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.798421 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-service-ca\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.798490 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.798478 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d79a012-9886-4c90-8286-f885ae0adcab-console-oauth-config\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.798531 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.798522 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d79a012-9886-4c90-8286-f885ae0adcab-console-serving-cert\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.851544 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.851498 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:08.899266 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.899239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-trusted-ca-bundle\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.899389 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.899287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-service-ca\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.899389 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.899334 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d79a012-9886-4c90-8286-f885ae0adcab-console-oauth-config\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.899844 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.899351 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d79a012-9886-4c90-8286-f885ae0adcab-console-serving-cert\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.900023 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.899991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-console-config\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.900149 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.900074 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvdz\" (UniqueName: \"kubernetes.io/projected/3d79a012-9886-4c90-8286-f885ae0adcab-kube-api-access-8gvdz\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.900149 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.900127 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-oauth-serving-cert\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.907833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.900620 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-trusted-ca-bundle\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.907833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.900963 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-service-ca\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.907833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.901371 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-console-config\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.907833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.902809 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d79a012-9886-4c90-8286-f885ae0adcab-console-oauth-config\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.907833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.903134 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d79a012-9886-4c90-8286-f885ae0adcab-console-serving-cert\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.907833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.907822 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d79a012-9886-4c90-8286-f885ae0adcab-oauth-serving-cert\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.914843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.914823 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvdz\" (UniqueName: \"kubernetes.io/projected/3d79a012-9886-4c90-8286-f885ae0adcab-kube-api-access-8gvdz\") pod \"console-6bc8d79ddd-l9hm4\" (UID: \"3d79a012-9886-4c90-8286-f885ae0adcab\") " pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:08.944841 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:08.944813 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:09.047330 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:09.047238 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:09.075134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:09.075109 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:09.213938 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:09.213907 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bc8d79ddd-l9hm4"] Aug 04 16:34:09.215156 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:34:09.215131 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d79a012_9886_4c90_8286_f885ae0adcab.slice/crio-2f8d54630462c6ee6f1bca773565e83e6174a2b3891881e54ae029bf84bc14fc WatchSource:0}: Error finding container 2f8d54630462c6ee6f1bca773565e83e6174a2b3891881e54ae029bf84bc14fc: Status 404 returned error can't find the container with id 2f8d54630462c6ee6f1bca773565e83e6174a2b3891881e54ae029bf84bc14fc Aug 04 16:34:09.514323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:09.514283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:10.164294 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:10.164250 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc8d79ddd-l9hm4" event={"ID":"3d79a012-9886-4c90-8286-f885ae0adcab","Type":"ContainerStarted","Data":"de70924b351f68a297f403f9ff13b49968a0fb1c2d8e8b0b4afe91f6331017ae"} Aug 04 16:34:10.164724 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:10.164314 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bc8d79ddd-l9hm4" event={"ID":"3d79a012-9886-4c90-8286-f885ae0adcab","Type":"ContainerStarted","Data":"2f8d54630462c6ee6f1bca773565e83e6174a2b3891881e54ae029bf84bc14fc"} Aug 04 16:34:10.187862 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:10.187808 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bc8d79ddd-l9hm4" podStartSLOduration=2.187793147 podStartE2EDuration="2.187793147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:34:10.185625161 +0000 UTC m=+2259.279225411" watchObservedRunningTime="2026-08-04 16:34:10.187793147 +0000 UTC m=+2259.281393395" Aug 04 16:34:17.375020 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:17.374980 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:19.075876 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:19.075845 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:19.076337 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:19.075901 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:19.080727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:19.080706 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:19.219790 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:19.219763 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bc8d79ddd-l9hm4" Aug 04 16:34:19.269385 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:19.269347 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6967b9f947-qkdkm"] Aug 04 16:34:19.382391 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:19.382316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:19.441113 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:19.441072 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:23.784782 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:23.784745 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:23.889330 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:23.889280 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:23.982477 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:23.982440 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:24.614776 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:24.614743 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:24.713013 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:24.712978 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:24.813123 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:24.813092 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:26.389623 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:26.389574 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:30.689311 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:30.689274 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:34:32.415823 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:32.415786 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:32.517767 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:32.517720 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:33.101385 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:33.101351 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:35.088957 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:35.088918 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:34:40.060975 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:40.060933 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:40.161667 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:40.161615 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:40.327324 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:40.327224 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:40.635056 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:40.634968 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:40.749384 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:40.749335 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:43.510760 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:43.510729 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:43.597125 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:43.597093 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:44.295236 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.295190 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6967b9f947-qkdkm" podUID="345e2a49-35f6-4433-8969-7cb7b593add3" containerName="console" containerID="cri-o://5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10" gracePeriod=15 Aug 04 16:34:44.543314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.543289 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:34:44.631441 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631408 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-trusted-ca-bundle\") pod \"345e2a49-35f6-4433-8969-7cb7b593add3\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " Aug 04 16:34:44.631625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631459 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-service-ca\") pod \"345e2a49-35f6-4433-8969-7cb7b593add3\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " Aug 04 16:34:44.631625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631498 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-console-config\") pod \"345e2a49-35f6-4433-8969-7cb7b593add3\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " Aug 04 16:34:44.631625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631514 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-oauth-serving-cert\") pod \"345e2a49-35f6-4433-8969-7cb7b593add3\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " Aug 04 16:34:44.631625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631568 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-serving-cert\") pod \"345e2a49-35f6-4433-8969-7cb7b593add3\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " Aug 04 16:34:44.631625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631614 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqcth\" (UniqueName: \"kubernetes.io/projected/345e2a49-35f6-4433-8969-7cb7b593add3-kube-api-access-hqcth\") pod \"345e2a49-35f6-4433-8969-7cb7b593add3\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " Aug 04 16:34:44.631904 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631663 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-oauth-config\") pod \"345e2a49-35f6-4433-8969-7cb7b593add3\" (UID: \"345e2a49-35f6-4433-8969-7cb7b593add3\") " Aug 04 16:34:44.632013 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.631970 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "345e2a49-35f6-4433-8969-7cb7b593add3" (UID: "345e2a49-35f6-4433-8969-7cb7b593add3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:34:44.632075 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.632005 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-service-ca" (OuterVolumeSpecName: "service-ca") pod "345e2a49-35f6-4433-8969-7cb7b593add3" (UID: "345e2a49-35f6-4433-8969-7cb7b593add3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:34:44.632112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.632090 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-console-config" (OuterVolumeSpecName: "console-config") pod "345e2a49-35f6-4433-8969-7cb7b593add3" (UID: "345e2a49-35f6-4433-8969-7cb7b593add3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:34:44.632374 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.632341 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "345e2a49-35f6-4433-8969-7cb7b593add3" (UID: "345e2a49-35f6-4433-8969-7cb7b593add3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:34:44.634146 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.634111 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/345e2a49-35f6-4433-8969-7cb7b593add3-kube-api-access-hqcth" (OuterVolumeSpecName: "kube-api-access-hqcth") pod "345e2a49-35f6-4433-8969-7cb7b593add3" (UID: "345e2a49-35f6-4433-8969-7cb7b593add3"). InnerVolumeSpecName "kube-api-access-hqcth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:34:44.634487 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.634459 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "345e2a49-35f6-4433-8969-7cb7b593add3" (UID: "345e2a49-35f6-4433-8969-7cb7b593add3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:34:44.634487 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.634478 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "345e2a49-35f6-4433-8969-7cb7b593add3" (UID: "345e2a49-35f6-4433-8969-7cb7b593add3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:34:44.645476 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.645446 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:44.732784 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.732752 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-console-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:34:44.732784 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.732777 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-oauth-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:34:44.732784 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.732788 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-serving-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:34:44.732998 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.732798 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcth\" (UniqueName: \"kubernetes.io/projected/345e2a49-35f6-4433-8969-7cb7b593add3-kube-api-access-hqcth\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:34:44.732998 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.732807 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/345e2a49-35f6-4433-8969-7cb7b593add3-console-oauth-config\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:34:44.732998 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.732815 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-trusted-ca-bundle\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:34:44.732998 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.732823 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/345e2a49-35f6-4433-8969-7cb7b593add3-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:34:44.840436 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:44.840396 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:45.372580 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.372546 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:45.373183 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.373161 2571 generic.go:356] "Generic (PLEG): container finished" podID="345e2a49-35f6-4433-8969-7cb7b593add3" containerID="5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10" exitCode=0 Aug 04 16:34:45.373313 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.373227 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6967b9f947-qkdkm" Aug 04 16:34:45.373313 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.373242 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967b9f947-qkdkm" event={"ID":"345e2a49-35f6-4433-8969-7cb7b593add3","Type":"ContainerDied","Data":"5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10"} Aug 04 16:34:45.373313 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.373288 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6967b9f947-qkdkm" event={"ID":"345e2a49-35f6-4433-8969-7cb7b593add3","Type":"ContainerDied","Data":"4391d406dea451850dccf13b177f999690d9998c9fa56e75ffd4399957348ba4"} Aug 04 16:34:45.373313 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.373310 2571 scope.go:117] "RemoveContainer" containerID="5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10" Aug 04 16:34:45.382743 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.382261 2571 scope.go:117] "RemoveContainer" containerID="5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10" Aug 04 16:34:45.382743 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:34:45.382493 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10\": container with ID starting with 5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10 not found: ID does not exist" containerID="5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10" Aug 04 16:34:45.382743 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.382515 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10"} err="failed to get container status \"5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10\": rpc error: code = NotFound desc = could not find container \"5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10\": container with ID starting with 5c52d5f0b6f4001138538df272a15aaa4e9d160ff5d7c9f7b1d46aa16ec6df10 not found: ID does not exist" Aug 04 16:34:45.408135 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.408106 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6967b9f947-qkdkm"] Aug 04 16:34:45.412452 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.412431 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6967b9f947-qkdkm"] Aug 04 16:34:45.469445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:45.469411 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:34:47.363446 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:47.363407 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="345e2a49-35f6-4433-8969-7cb7b593add3" path="/var/lib/kubelet/pods/345e2a49-35f6-4433-8969-7cb7b593add3/volumes" Aug 04 16:34:48.108199 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:34:48.108161 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:35:08.555259 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:08.555199 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:09.288753 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:09.288719 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:09.369359 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:09.369327 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:09.742178 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:09.742130 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:15.677635 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:15.677594 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:15.795261 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:15.795226 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:16.054764 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.054733 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj"] Aug 04 16:35:16.055138 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.055125 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="345e2a49-35f6-4433-8969-7cb7b593add3" containerName="console" Aug 04 16:35:16.055192 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.055140 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="345e2a49-35f6-4433-8969-7cb7b593add3" containerName="console" Aug 04 16:35:16.055252 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.055212 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="345e2a49-35f6-4433-8969-7cb7b593add3" containerName="console" Aug 04 16:35:16.058708 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.058690 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" Aug 04 16:35:16.061771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.061753 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:35:16.061938 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.061915 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:35:16.072408 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.072385 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj"] Aug 04 16:35:16.128933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.128897 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c0cf70-d165-4183-aafd-2f0f4478c8a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj\" (UID: \"61c0cf70-d165-4183-aafd-2f0f4478c8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" Aug 04 16:35:16.230220 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.230173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c0cf70-d165-4183-aafd-2f0f4478c8a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj\" (UID: \"61c0cf70-d165-4183-aafd-2f0f4478c8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" Aug 04 16:35:16.230810 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.230788 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c0cf70-d165-4183-aafd-2f0f4478c8a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj\" (UID: \"61c0cf70-d165-4183-aafd-2f0f4478c8a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" Aug 04 16:35:16.369268 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.369166 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" Aug 04 16:35:16.517649 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.517622 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj"] Aug 04 16:35:16.519702 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:35:16.519665 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c0cf70_d165_4183_aafd_2f0f4478c8a0.slice/crio-029643821a171e0c2034c0f1212b5ccbccbad804f66928056c0ae2894b8a7135 WatchSource:0}: Error finding container 029643821a171e0c2034c0f1212b5ccbccbad804f66928056c0ae2894b8a7135: Status 404 returned error can't find the container with id 029643821a171e0c2034c0f1212b5ccbccbad804f66928056c0ae2894b8a7135 Aug 04 16:35:16.556994 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:16.556960 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" event={"ID":"61c0cf70-d165-4183-aafd-2f0f4478c8a0","Type":"ContainerStarted","Data":"029643821a171e0c2034c0f1212b5ccbccbad804f66928056c0ae2894b8a7135"} Aug 04 16:35:17.564436 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:17.564404 2571 generic.go:356] "Generic (PLEG): container finished" podID="61c0cf70-d165-4183-aafd-2f0f4478c8a0" containerID="a715a3ac13c7acac2fb9da1d103a839bcbb9c6fd61ed5581af6a9306ac66f168" exitCode=0 Aug 04 16:35:17.564854 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:17.564465 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" event={"ID":"61c0cf70-d165-4183-aafd-2f0f4478c8a0","Type":"ContainerDied","Data":"a715a3ac13c7acac2fb9da1d103a839bcbb9c6fd61ed5581af6a9306ac66f168"} Aug 04 16:35:18.702503 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:18.702477 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" Aug 04 16:35:18.753293 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:18.753263 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c0cf70-d165-4183-aafd-2f0f4478c8a0-maas-api-service-ca\") pod \"61c0cf70-d165-4183-aafd-2f0f4478c8a0\" (UID: \"61c0cf70-d165-4183-aafd-2f0f4478c8a0\") " Aug 04 16:35:18.753620 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:18.753596 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c0cf70-d165-4183-aafd-2f0f4478c8a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61c0cf70-d165-4183-aafd-2f0f4478c8a0" (UID: "61c0cf70-d165-4183-aafd-2f0f4478c8a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:35:18.854703 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:18.854628 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c0cf70-d165-4183-aafd-2f0f4478c8a0-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:35:19.577991 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:19.577958 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" Aug 04 16:35:19.578162 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:19.577995 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj" event={"ID":"61c0cf70-d165-4183-aafd-2f0f4478c8a0","Type":"ContainerDied","Data":"029643821a171e0c2034c0f1212b5ccbccbad804f66928056c0ae2894b8a7135"} Aug 04 16:35:19.578162 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:19.578069 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="029643821a171e0c2034c0f1212b5ccbccbad804f66928056c0ae2894b8a7135" Aug 04 16:35:20.441141 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:20.441108 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:27.037266 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:27.037166 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj"] Aug 04 16:35:27.039536 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:27.039516 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7c9a38ca-nlkvj"] Aug 04 16:35:27.363565 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:27.363536 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c0cf70-d165-4183-aafd-2f0f4478c8a0" path="/var/lib/kubelet/pods/61c0cf70-d165-4183-aafd-2f0f4478c8a0/volumes" Aug 04 16:35:27.436144 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:27.436105 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:27.541007 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:27.540958 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:28.998546 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:28.998510 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:35:33.654404 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:33.654368 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:33.880538 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:33.880502 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:34.022636 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:34.022532 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:34.264951 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:34.264908 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:34.427053 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:34.427000 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:34.518417 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:34.518387 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:34.620155 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:34.620121 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:41.154101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:41.154065 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:41.667739 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:41.667701 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:41.771894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:41.771859 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:47.753926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:47.753878 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:35:47.832127 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:35:47.832086 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:36:09.427300 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:09.427266 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:36:16.497398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.497356 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq"] Aug 04 16:36:16.497887 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.497568 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" podUID="50eddab3-9851-49f9-9121-c184c7435c26" containerName="manager" containerID="cri-o://199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe" gracePeriod=10 Aug 04 16:36:16.742589 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.742561 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:36:16.880623 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.880589 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/50eddab3-9851-49f9-9121-c184c7435c26-kube-api-access-s4nfg\") pod \"50eddab3-9851-49f9-9121-c184c7435c26\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " Aug 04 16:36:16.880830 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.880702 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50eddab3-9851-49f9-9121-c184c7435c26-extensions-socket-volume\") pod \"50eddab3-9851-49f9-9121-c184c7435c26\" (UID: \"50eddab3-9851-49f9-9121-c184c7435c26\") " Aug 04 16:36:16.881111 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.881085 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eddab3-9851-49f9-9121-c184c7435c26-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "50eddab3-9851-49f9-9121-c184c7435c26" (UID: "50eddab3-9851-49f9-9121-c184c7435c26"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:36:16.882718 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.882697 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eddab3-9851-49f9-9121-c184c7435c26-kube-api-access-s4nfg" (OuterVolumeSpecName: "kube-api-access-s4nfg") pod "50eddab3-9851-49f9-9121-c184c7435c26" (UID: "50eddab3-9851-49f9-9121-c184c7435c26"). InnerVolumeSpecName "kube-api-access-s4nfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:36:16.918509 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.918475 2571 generic.go:356] "Generic (PLEG): container finished" podID="50eddab3-9851-49f9-9121-c184c7435c26" containerID="199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe" exitCode=0 Aug 04 16:36:16.918669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.918519 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" Aug 04 16:36:16.918669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.918560 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" event={"ID":"50eddab3-9851-49f9-9121-c184c7435c26","Type":"ContainerDied","Data":"199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe"} Aug 04 16:36:16.918669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.918616 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq" event={"ID":"50eddab3-9851-49f9-9121-c184c7435c26","Type":"ContainerDied","Data":"a7c9552c9f2727842cf7a86b5e5821e61f695c09f0a7f4b8012584af94a81b72"} Aug 04 16:36:16.918669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.918647 2571 scope.go:117] "RemoveContainer" containerID="199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe" Aug 04 16:36:16.930125 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.930105 2571 scope.go:117] "RemoveContainer" containerID="199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe" Aug 04 16:36:16.930458 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:36:16.930433 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe\": container with ID starting with 199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe not found: ID does not exist" containerID="199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe" Aug 04 16:36:16.930551 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.930466 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe"} err="failed to get container status \"199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe\": rpc error: code = NotFound desc = could not find container \"199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe\": container with ID starting with 199e9f021265914050f6b7811e9546dcd1a632fdc9348c4fed30a63f98a1aebe not found: ID does not exist" Aug 04 16:36:16.945224 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.945172 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq"] Aug 04 16:36:16.947408 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.947390 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xd8hq"] Aug 04 16:36:16.981952 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.981914 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4nfg\" (UniqueName: \"kubernetes.io/projected/50eddab3-9851-49f9-9121-c184c7435c26-kube-api-access-s4nfg\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:36:16.981952 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:16.981948 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/50eddab3-9851-49f9-9121-c184c7435c26-extensions-socket-volume\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:36:17.362774 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:17.362741 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50eddab3-9851-49f9-9121-c184c7435c26" path="/var/lib/kubelet/pods/50eddab3-9851-49f9-9121-c184c7435c26/volumes" Aug 04 16:36:39.392237 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:39.392172 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:36:39.521187 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:36:39.521146 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:31.698876 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:31.698836 2571 scope.go:117] "RemoveContainer" containerID="28b56d687f38e93f4bde83de1af3fa7ef6c5900076245d750bdfd642b1bfac5a" Aug 04 16:37:31.708323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:31.708294 2571 scope.go:117] "RemoveContainer" containerID="5026e15d3f1b8fd41f11d6edc5fbdef5df77e393be33f1b81561bc8aeb989aad" Aug 04 16:37:32.155270 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.155226 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb"] Aug 04 16:37:32.155644 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.155629 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50eddab3-9851-49f9-9121-c184c7435c26" containerName="manager" Aug 04 16:37:32.155644 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.155644 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="50eddab3-9851-49f9-9121-c184c7435c26" containerName="manager" Aug 04 16:37:32.155771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.155663 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c0cf70-d165-4183-aafd-2f0f4478c8a0" containerName="revoke-keys" Aug 04 16:37:32.155771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.155668 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c0cf70-d165-4183-aafd-2f0f4478c8a0" containerName="revoke-keys" Aug 04 16:37:32.155771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.155733 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="50eddab3-9851-49f9-9121-c184c7435c26" containerName="manager" Aug 04 16:37:32.155771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.155742 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c0cf70-d165-4183-aafd-2f0f4478c8a0" containerName="revoke-keys" Aug 04 16:37:32.160406 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.160384 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.163681 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.163663 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-vhhgd\"" Aug 04 16:37:32.171830 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.171807 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb"] Aug 04 16:37:32.256421 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.256379 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wr2\" (UniqueName: \"kubernetes.io/projected/90973b46-253c-4e53-aac8-ed789fd9c0dd-kube-api-access-v6wr2\") pod \"kuadrant-operator-controller-manager-68cf589589-m2ksb\" (UID: \"90973b46-253c-4e53-aac8-ed789fd9c0dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.256606 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.256461 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90973b46-253c-4e53-aac8-ed789fd9c0dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-m2ksb\" (UID: \"90973b46-253c-4e53-aac8-ed789fd9c0dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.357363 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.357329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wr2\" (UniqueName: \"kubernetes.io/projected/90973b46-253c-4e53-aac8-ed789fd9c0dd-kube-api-access-v6wr2\") pod \"kuadrant-operator-controller-manager-68cf589589-m2ksb\" (UID: \"90973b46-253c-4e53-aac8-ed789fd9c0dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.357527 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.357387 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90973b46-253c-4e53-aac8-ed789fd9c0dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-m2ksb\" (UID: \"90973b46-253c-4e53-aac8-ed789fd9c0dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.357745 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.357729 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90973b46-253c-4e53-aac8-ed789fd9c0dd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-m2ksb\" (UID: \"90973b46-253c-4e53-aac8-ed789fd9c0dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.366725 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.366697 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wr2\" (UniqueName: \"kubernetes.io/projected/90973b46-253c-4e53-aac8-ed789fd9c0dd-kube-api-access-v6wr2\") pod \"kuadrant-operator-controller-manager-68cf589589-m2ksb\" (UID: \"90973b46-253c-4e53-aac8-ed789fd9c0dd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.471620 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.471533 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:32.611727 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:32.611698 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb"] Aug 04 16:37:32.612742 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:37:32.612717 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90973b46_253c_4e53_aac8_ed789fd9c0dd.slice/crio-8ca2198e40a3d389a59b0ac9c9cbb2470d9ad511399e5553ce9cc9ec9e00a2e9 WatchSource:0}: Error finding container 8ca2198e40a3d389a59b0ac9c9cbb2470d9ad511399e5553ce9cc9ec9e00a2e9: Status 404 returned error can't find the container with id 8ca2198e40a3d389a59b0ac9c9cbb2470d9ad511399e5553ce9cc9ec9e00a2e9 Aug 04 16:37:33.345341 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:33.345299 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" event={"ID":"90973b46-253c-4e53-aac8-ed789fd9c0dd","Type":"ContainerStarted","Data":"ad6f4d48373929235d25cd927f22e054dcae825c50c9240f87bfa0eb664ea431"} Aug 04 16:37:33.345828 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:33.345393 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" event={"ID":"90973b46-253c-4e53-aac8-ed789fd9c0dd","Type":"ContainerStarted","Data":"8ca2198e40a3d389a59b0ac9c9cbb2470d9ad511399e5553ce9cc9ec9e00a2e9"} Aug 04 16:37:33.345828 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:33.345449 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:33.395672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:33.395616 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" podStartSLOduration=1.3956008469999999 podStartE2EDuration="1.395600847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:37:33.392738476 +0000 UTC m=+2462.486338723" watchObservedRunningTime="2026-08-04 16:37:33.395600847 +0000 UTC m=+2462.489201094" Aug 04 16:37:44.354331 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:44.354299 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-m2ksb" Aug 04 16:37:51.030588 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:51.030546 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:51.095953 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:51.095917 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:37:51.388129 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:51.388094 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:51.537030 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:51.536997 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:54.314042 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:54.314004 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:54.577399 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:54.577311 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:54.729682 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:54.729641 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:55.709395 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:55.709354 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:56.128690 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:56.128649 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:56.251439 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:56.251405 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:56.355373 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:56.355338 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:37:58.400405 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:37:58.400333 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:38:19.458057 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:19.458022 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:38:26.989894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:26.989811 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4"] Aug 04 16:38:26.993517 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:26.993494 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:26.996577 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:26.996554 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c2a82a29\"/\"openshift-service-ca.crt\"" Aug 04 16:38:26.997833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:26.997815 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c2a82a29\"/\"kube-root-ca.crt\"" Aug 04 16:38:26.997833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:26.997823 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c2a82a29\"/\"e2e-lifecycle-model-c2a82a29-kserve-self-signed-certs\"" Aug 04 16:38:26.997996 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:26.997845 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c2a82a29\"/\"default-dockercfg-gnw6l\"" Aug 04 16:38:27.005153 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.005122 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4"] Aug 04 16:38:27.022556 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.022521 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:38:27.064796 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.064760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-dshm\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.064987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.064807 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcpxc\" (UniqueName: \"kubernetes.io/projected/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-kube-api-access-pcpxc\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.064987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.064839 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-model-cache\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.064987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.064908 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tls-certs\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.064987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.064947 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tmp-dir\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.064987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.064973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-home\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.165809 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.165779 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcpxc\" (UniqueName: \"kubernetes.io/projected/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-kube-api-access-pcpxc\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166006 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.165815 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-model-cache\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166006 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.165864 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tls-certs\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166006 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.165897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tmp-dir\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166006 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.165916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-home\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166006 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.165992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-dshm\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166406 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.166379 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tmp-dir\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166480 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.166418 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-model-cache\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.166480 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.166442 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-home\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.168270 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.168247 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-dshm\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.168556 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.168537 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tls-certs\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.174213 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.174173 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcpxc\" (UniqueName: \"kubernetes.io/projected/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-kube-api-access-pcpxc\") pod \"e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.302750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.302652 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:27.444793 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.444763 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4"] Aug 04 16:38:27.446681 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:38:27.446653 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ee58e8c_af4e_4108_be0e_0ceb5a7c89fd.slice/crio-936a8430c6b4fca06f17d2a4f3f8cf4510f621788098c70632573a5b888e0613 WatchSource:0}: Error finding container 936a8430c6b4fca06f17d2a4f3f8cf4510f621788098c70632573a5b888e0613: Status 404 returned error can't find the container with id 936a8430c6b4fca06f17d2a4f3f8cf4510f621788098c70632573a5b888e0613 Aug 04 16:38:27.652335 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.652296 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" event={"ID":"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd","Type":"ContainerStarted","Data":"8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3"} Aug 04 16:38:27.652533 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.652350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" event={"ID":"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd","Type":"ContainerStarted","Data":"936a8430c6b4fca06f17d2a4f3f8cf4510f621788098c70632573a5b888e0613"} Aug 04 16:38:27.674809 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:27.674756 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" podStartSLOduration=1.674739709 podStartE2EDuration="1.674739709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:38:27.672119343 +0000 UTC m=+2516.765719592" watchObservedRunningTime="2026-08-04 16:38:27.674739709 +0000 UTC m=+2516.768339957" Aug 04 16:38:28.490661 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:28.490626 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:38:31.732805 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:31.732774 2571 scope.go:117] "RemoveContainer" containerID="856a6ea7a5e5225f85d095128ca148cd74d0ae2b7282ac0c05a845b01ae697cb" Aug 04 16:38:37.303340 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:37.303301 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:37.303749 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:37.303354 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:37.315808 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:37.315780 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:37.721788 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:37.721752 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:38:47.455321 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:47.455283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:38:47.554461 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:47.554425 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:38:51.512329 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:38:51.512285 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:39:12.501723 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:12.501685 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:39:14.860761 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:14.860726 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:39:16.567781 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:16.567737 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:39:18.325243 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:18.325194 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:39:20.288921 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:20.288883 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:39:23.301742 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:23.301702 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4"] Aug 04 16:39:23.399639 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:23.399608 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:39:31.756094 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:31.756062 2571 scope.go:117] "RemoveContainer" containerID="348abde6f98be7ac2740bd9ebd1dc9c1d853e8218fb8138cd3f2aa7302190461" Aug 04 16:39:38.321106 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.321059 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" podUID="5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" containerName="main" containerID="cri-o://8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3" gracePeriod=45 Aug 04 16:39:38.579070 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.579005 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:39:38.623635 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.623601 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-model-cache\") pod \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " Aug 04 16:39:38.623789 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.623699 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcpxc\" (UniqueName: \"kubernetes.io/projected/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-kube-api-access-pcpxc\") pod \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " Aug 04 16:39:38.623789 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.623765 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-home\") pod \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " Aug 04 16:39:38.623922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.623810 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tmp-dir\") pod \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " Aug 04 16:39:38.623922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.623845 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tls-certs\") pod \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " Aug 04 16:39:38.623922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.623874 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-dshm\") pod \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\" (UID: \"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd\") " Aug 04 16:39:38.624128 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.624079 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" (UID: "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:39:38.624284 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.624092 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-home" (OuterVolumeSpecName: "home") pod "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" (UID: "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:39:38.624284 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.624227 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:39:38.624750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.624630 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" (UID: "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:39:38.626553 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.626524 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-dshm" (OuterVolumeSpecName: "dshm") pod "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" (UID: "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:39:38.626553 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.626541 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-kube-api-access-pcpxc" (OuterVolumeSpecName: "kube-api-access-pcpxc") pod "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" (UID: "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd"). InnerVolumeSpecName "kube-api-access-pcpxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:39:38.626878 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.626863 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" (UID: "5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:39:38.725098 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.725065 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:39:38.725098 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.725093 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcpxc\" (UniqueName: \"kubernetes.io/projected/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-kube-api-access-pcpxc\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:39:38.725098 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.725106 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:39:38.725358 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.725115 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:39:38.725358 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:38.725124 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:39:39.053318 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.053196 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" containerID="8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3" exitCode=0 Aug 04 16:39:39.053318 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.053251 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" Aug 04 16:39:39.053318 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.053302 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" event={"ID":"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd","Type":"ContainerDied","Data":"8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3"} Aug 04 16:39:39.053529 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.053344 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4" event={"ID":"5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd","Type":"ContainerDied","Data":"936a8430c6b4fca06f17d2a4f3f8cf4510f621788098c70632573a5b888e0613"} Aug 04 16:39:39.053529 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.053366 2571 scope.go:117] "RemoveContainer" containerID="8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3" Aug 04 16:39:39.062546 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.062529 2571 scope.go:117] "RemoveContainer" containerID="8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3" Aug 04 16:39:39.062806 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:39:39.062784 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3\": container with ID starting with 8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3 not found: ID does not exist" containerID="8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3" Aug 04 16:39:39.062847 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.062815 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3"} err="failed to get container status \"8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3\": rpc error: code = NotFound desc = could not find container \"8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3\": container with ID starting with 8d5f7ac41366ea71c613f8baed073dc63f8d2e50c08cb685bc17db95b52045f3 not found: ID does not exist" Aug 04 16:39:39.077686 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.077665 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4"] Aug 04 16:39:39.079844 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.079824 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c2a82a29/e2e-lifecycle-model-c2a82a29-kserve-7469c5d9ff-ssdc4"] Aug 04 16:39:39.365913 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:39.365871 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" path="/var/lib/kubelet/pods/5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd/volumes" Aug 04 16:39:59.626263 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:39:59.626174 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:00.541982 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:00.541940 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:06.549328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:06.549286 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:06.663810 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:06.663776 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:07.066380 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:07.066346 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:07.167123 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:07.167091 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:08.692245 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:08.692195 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:40:11.755022 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:11.754980 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:18.420615 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:18.420577 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:18.508826 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:18.508788 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:19.245037 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:19.244989 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:19.371987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:19.371950 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:19.470168 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:19.470123 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:20.833165 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:20.833116 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:21.023669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:21.023633 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:40:28.031466 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:28.031428 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:40:28.237288 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:28.237241 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:28.428050 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:28.428012 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:28.958852 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:28.958797 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:35.602788 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:35.602752 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:35.717417 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:35.717386 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:36.038390 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:36.038302 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:36.500218 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:36.500180 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:40:46.380575 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:46.380522 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:40:46.625571 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:40:46.625532 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:07.671474 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:07.671431 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:14.120272 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:14.120240 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:14.252989 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:14.252944 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:14.514699 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:14.514607 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:14.836241 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:14.836193 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:15.093163 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:15.093074 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:17.493688 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:17.493654 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:17.600030 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:17.599993 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:18.622905 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:18.622853 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:18.776306 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:18.776267 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:19.414936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:19.414905 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:19.634975 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:19.634929 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:22.229369 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:22.229337 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:41:30.449351 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:30.449244 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:31.789130 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:31.789099 2571 scope.go:117] "RemoveContainer" containerID="a715a3ac13c7acac2fb9da1d103a839bcbb9c6fd61ed5581af6a9306ac66f168" Aug 04 16:41:42.520942 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:42.520847 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:42.865792 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:42.865756 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:43.260132 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:43.260046 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:43.456033 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:43.455999 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:46.641773 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:46.641731 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:46.736172 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:46.736134 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:47.915110 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:47.915073 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:48.033712 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:48.033676 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:48.474116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:48.474085 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:48.571464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:48.571426 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:41:51.198616 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:41:51.198571 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:42:11.621172 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:11.621130 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:42:15.396882 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.396839 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg"] Aug 04 16:42:15.397453 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.397431 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" containerName="main" Aug 04 16:42:15.397529 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.397457 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" containerName="main" Aug 04 16:42:15.397592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.397570 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ee58e8c-af4e-4108-be0e-0ceb5a7c89fd" containerName="main" Aug 04 16:42:15.400447 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.400423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.403610 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.403587 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df4f93\"/\"kube-root-ca.crt\"" Aug 04 16:42:15.404772 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.404755 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df4f93\"/\"default-dockercfg-b7crh\"" Aug 04 16:42:15.404772 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.404765 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df4f93\"/\"openshift-service-ca.crt\"" Aug 04 16:42:15.405051 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.405035 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df4f93\"/\"test-model-df4f93-kserve-self-signed-certs\"" Aug 04 16:42:15.414319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.414286 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg"] Aug 04 16:42:15.435660 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.435628 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:42:15.593047 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.593009 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tmp-dir\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.593047 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.593058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tls-certs\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.593347 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.593191 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-model-cache\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.593347 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.593306 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xszq\" (UniqueName: \"kubernetes.io/projected/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-kube-api-access-7xszq\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.593436 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.593356 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-home\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.593436 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.593375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-dshm\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694416 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694318 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-model-cache\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694416 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xszq\" (UniqueName: \"kubernetes.io/projected/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-kube-api-access-7xszq\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694633 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-home\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694633 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694459 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-dshm\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694633 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694485 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tmp-dir\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694633 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694515 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tls-certs\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694782 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-model-cache\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694860 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694822 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-home\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.694916 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.694899 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tmp-dir\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.696858 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.696835 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-dshm\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.697113 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.697096 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tls-certs\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.704540 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.704504 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xszq\" (UniqueName: \"kubernetes.io/projected/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-kube-api-access-7xszq\") pod \"test-model-df4f93-kserve-5d64f885b9-4gqgg\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.710338 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.710308 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:15.860985 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.860949 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg"] Aug 04 16:42:15.861284 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:42:15.861262 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea9f8d3b_084f_467d_99ce_f59ecc4bb61b.slice/crio-d4f6ae11dd13d3590b690660455076645ad97068f0556e6807bac1bce110775b WatchSource:0}: Error finding container d4f6ae11dd13d3590b690660455076645ad97068f0556e6807bac1bce110775b: Status 404 returned error can't find the container with id d4f6ae11dd13d3590b690660455076645ad97068f0556e6807bac1bce110775b Aug 04 16:42:15.933839 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.933803 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" event={"ID":"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b","Type":"ContainerStarted","Data":"fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4"} Aug 04 16:42:15.934012 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.933852 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" event={"ID":"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b","Type":"ContainerStarted","Data":"d4f6ae11dd13d3590b690660455076645ad97068f0556e6807bac1bce110775b"} Aug 04 16:42:15.956343 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:15.956195 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" podStartSLOduration=0.956173965 podStartE2EDuration="956.173965ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:42:15.95473945 +0000 UTC m=+2745.048339700" watchObservedRunningTime="2026-08-04 16:42:15.956173965 +0000 UTC m=+2745.049774215" Aug 04 16:42:17.907856 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:17.907813 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:42:25.279545 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:25.279508 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:42:25.711339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:25.711283 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:25.711339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:25.711349 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:25.725583 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:25.725553 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:26.001322 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:26.001237 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:42:50.087218 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.087165 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr"] Aug 04 16:42:50.091625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.091597 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.095685 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.095183 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1f0f0c\"/\"kube-root-ca.crt\"" Aug 04 16:42:50.096028 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.095997 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1f0f0c\"/\"openshift-service-ca.crt\"" Aug 04 16:42:50.096175 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.096057 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1f0f0c\"/\"test-model-1f0f0c-kserve-self-signed-certs\"" Aug 04 16:42:50.096266 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.096235 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1f0f0c\"/\"default-dockercfg-8g5hs\"" Aug 04 16:42:50.104039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.104006 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr"] Aug 04 16:42:50.138439 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.138387 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:42:50.216733 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.216691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-home\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.216926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.216772 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-tmp-dir\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.216926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.216794 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-model-cache\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.216926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.216828 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-dshm\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.217097 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.216927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1e8884-9f29-4658-8227-e43e6fab03c7-tls-certs\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.217097 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.216957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4t7\" (UniqueName: \"kubernetes.io/projected/db1e8884-9f29-4658-8227-e43e6fab03c7-kube-api-access-tr4t7\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318189 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-dshm\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318418 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318215 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1e8884-9f29-4658-8227-e43e6fab03c7-tls-certs\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318418 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4t7\" (UniqueName: \"kubernetes.io/projected/db1e8884-9f29-4658-8227-e43e6fab03c7-kube-api-access-tr4t7\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318418 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318261 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-home\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318418 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318319 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-tmp-dir\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318418 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318338 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-model-cache\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318726 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318700 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-home\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318726 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318721 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-tmp-dir\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.318842 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.318788 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-model-cache\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.320581 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.320561 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-dshm\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.320751 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.320732 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1e8884-9f29-4658-8227-e43e6fab03c7-tls-certs\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.327825 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.327799 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4t7\" (UniqueName: \"kubernetes.io/projected/db1e8884-9f29-4658-8227-e43e6fab03c7-kube-api-access-tr4t7\") pod \"test-model-1f0f0c-kserve-6cfbb984cc-972tr\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.403521 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.403420 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:42:50.547226 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:50.547184 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr"] Aug 04 16:42:50.549799 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:42:50.549763 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb1e8884_9f29_4658_8227_e43e6fab03c7.slice/crio-1ee77c919aeb7560c7f28ed83cbc7c276f05dac0517dae2c1816199731c36c26 WatchSource:0}: Error finding container 1ee77c919aeb7560c7f28ed83cbc7c276f05dac0517dae2c1816199731c36c26: Status 404 returned error can't find the container with id 1ee77c919aeb7560c7f28ed83cbc7c276f05dac0517dae2c1816199731c36c26 Aug 04 16:42:51.135608 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:51.135571 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" event={"ID":"db1e8884-9f29-4658-8227-e43e6fab03c7","Type":"ContainerStarted","Data":"a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c"} Aug 04 16:42:51.135608 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:51.135613 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" event={"ID":"db1e8884-9f29-4658-8227-e43e6fab03c7","Type":"ContainerStarted","Data":"1ee77c919aeb7560c7f28ed83cbc7c276f05dac0517dae2c1816199731c36c26"} Aug 04 16:42:51.154566 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:51.154511 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" podStartSLOduration=1.154491666 podStartE2EDuration="1.154491666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:42:51.153928012 +0000 UTC m=+2780.247528272" watchObservedRunningTime="2026-08-04 16:42:51.154491666 +0000 UTC m=+2780.248092279" Aug 04 16:42:53.285492 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:42:53.285403 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:43:00.403665 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:00.403620 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:43:00.404149 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:00.403680 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:43:00.416437 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:00.416409 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:43:01.202910 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:01.202883 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:43:03.609038 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:03.608999 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:43:20.382506 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:20.382460 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:31.615133 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:31.615094 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg"] Aug 04 16:43:31.629316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:31.629271 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:32.552197 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.552158 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr"] Aug 04 16:43:32.565669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.565631 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:32.690414 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.690375 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt"] Aug 04 16:43:32.694051 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.694032 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" Aug 04 16:43:32.697001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.696978 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:43:32.697111 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.697039 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:43:32.702829 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.702797 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt"] Aug 04 16:43:32.847120 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.847087 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ecf361-190b-4028-80c8-00097d16d1e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt\" (UID: \"f6ecf361-190b-4028-80c8-00097d16d1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" Aug 04 16:43:32.948658 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.948616 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ecf361-190b-4028-80c8-00097d16d1e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt\" (UID: \"f6ecf361-190b-4028-80c8-00097d16d1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" Aug 04 16:43:32.949261 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:32.949242 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ecf361-190b-4028-80c8-00097d16d1e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt\" (UID: \"f6ecf361-190b-4028-80c8-00097d16d1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" Aug 04 16:43:33.005737 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:33.005696 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" Aug 04 16:43:33.149699 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:33.149670 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt"] Aug 04 16:43:33.152068 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:43:33.152034 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ecf361_190b_4028_80c8_00097d16d1e5.slice/crio-d801a91b06ac1119df1280c39ef1d05046ab872fdd5b86b965dba0397c774338 WatchSource:0}: Error finding container d801a91b06ac1119df1280c39ef1d05046ab872fdd5b86b965dba0397c774338: Status 404 returned error can't find the container with id d801a91b06ac1119df1280c39ef1d05046ab872fdd5b86b965dba0397c774338 Aug 04 16:43:33.371392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:33.371354 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" event={"ID":"f6ecf361-190b-4028-80c8-00097d16d1e5","Type":"ContainerStarted","Data":"d801a91b06ac1119df1280c39ef1d05046ab872fdd5b86b965dba0397c774338"} Aug 04 16:43:34.385603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:34.385564 2571 generic.go:356] "Generic (PLEG): container finished" podID="f6ecf361-190b-4028-80c8-00097d16d1e5" containerID="27be5f306e4e65c60e0815b6d495213491e6f8cf89e8f10e946bf52d8ca621db" exitCode=0 Aug 04 16:43:34.386058 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:34.385655 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" event={"ID":"f6ecf361-190b-4028-80c8-00097d16d1e5","Type":"ContainerDied","Data":"27be5f306e4e65c60e0815b6d495213491e6f8cf89e8f10e946bf52d8ca621db"} Aug 04 16:43:35.532046 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:35.532014 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" Aug 04 16:43:35.571482 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:35.571444 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ecf361-190b-4028-80c8-00097d16d1e5-maas-api-service-ca\") pod \"f6ecf361-190b-4028-80c8-00097d16d1e5\" (UID: \"f6ecf361-190b-4028-80c8-00097d16d1e5\") " Aug 04 16:43:35.571816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:35.571788 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ecf361-190b-4028-80c8-00097d16d1e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6ecf361-190b-4028-80c8-00097d16d1e5" (UID: "f6ecf361-190b-4028-80c8-00097d16d1e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:43:35.672759 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:35.672667 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ecf361-190b-4028-80c8-00097d16d1e5-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:36.398603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:36.398567 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" Aug 04 16:43:36.398603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:36.398592 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt" event={"ID":"f6ecf361-190b-4028-80c8-00097d16d1e5","Type":"ContainerDied","Data":"d801a91b06ac1119df1280c39ef1d05046ab872fdd5b86b965dba0397c774338"} Aug 04 16:43:36.398848 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:36.398656 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d801a91b06ac1119df1280c39ef1d05046ab872fdd5b86b965dba0397c774338" Aug 04 16:43:36.679449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:36.679351 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:43:43.346045 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:43.346008 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:43.412373 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:43.412338 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:43.511536 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:43.511495 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:43.536870 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:43.536833 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt"] Aug 04 16:43:43.540740 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:43.540715 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df4f93-vfzqt"] Aug 04 16:43:43.586784 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:43.586747 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:43.934410 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:43.934369 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:45.363927 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:45.363890 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ecf361-190b-4028-80c8-00097d16d1e5" path="/var/lib/kubelet/pods/f6ecf361-190b-4028-80c8-00097d16d1e5/volumes" Aug 04 16:43:45.487545 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:45.487507 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:43:46.636506 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.636463 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" podUID="ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" containerName="main" containerID="cri-o://fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4" gracePeriod=45 Aug 04 16:43:46.885106 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.885075 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:43:46.979031 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.978937 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tls-certs\") pod \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " Aug 04 16:43:46.979031 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979013 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-model-cache\") pod \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " Aug 04 16:43:46.979336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979038 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-dshm\") pod \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " Aug 04 16:43:46.979336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979080 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-home\") pod \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " Aug 04 16:43:46.979336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979113 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tmp-dir\") pod \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " Aug 04 16:43:46.979336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979158 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xszq\" (UniqueName: \"kubernetes.io/projected/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-kube-api-access-7xszq\") pod \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\" (UID: \"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b\") " Aug 04 16:43:46.979570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979339 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-home" (OuterVolumeSpecName: "home") pod "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" (UID: "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:46.979570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979354 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" (UID: "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:46.979570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979426 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" (UID: "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:46.979570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979506 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:46.979570 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.979520 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:46.981437 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.981416 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" (UID: "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:43:46.981536 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.981517 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-dshm" (OuterVolumeSpecName: "dshm") pod "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" (UID: "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:46.981584 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:46.981529 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-kube-api-access-7xszq" (OuterVolumeSpecName: "kube-api-access-7xszq") pod "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" (UID: "ea9f8d3b-084f-467d-99ce-f59ecc4bb61b"). InnerVolumeSpecName "kube-api-access-7xszq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:43:47.080108 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.080069 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.080108 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.080109 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.080373 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.080123 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.080373 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.080137 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xszq\" (UniqueName: \"kubernetes.io/projected/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b-kube-api-access-7xszq\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.462926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.462886 2571 generic.go:356] "Generic (PLEG): container finished" podID="ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" containerID="fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4" exitCode=0 Aug 04 16:43:47.463134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.462953 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" Aug 04 16:43:47.463134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.462972 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" event={"ID":"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b","Type":"ContainerDied","Data":"fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4"} Aug 04 16:43:47.463134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.463025 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg" event={"ID":"ea9f8d3b-084f-467d-99ce-f59ecc4bb61b","Type":"ContainerDied","Data":"d4f6ae11dd13d3590b690660455076645ad97068f0556e6807bac1bce110775b"} Aug 04 16:43:47.463134 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.463047 2571 scope.go:117] "RemoveContainer" containerID="fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4" Aug 04 16:43:47.473362 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.473340 2571 scope.go:117] "RemoveContainer" containerID="fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4" Aug 04 16:43:47.473668 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:43:47.473645 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4\": container with ID starting with fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4 not found: ID does not exist" containerID="fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4" Aug 04 16:43:47.473739 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.473681 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4"} err="failed to get container status \"fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4\": rpc error: code = NotFound desc = could not find container \"fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4\": container with ID starting with fc1afa6f9104459260246d080774c84b1a2079e224c461d7cbe976600a1e73b4 not found: ID does not exist" Aug 04 16:43:47.480846 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.480819 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg"] Aug 04 16:43:47.486910 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.486886 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-df4f93/test-model-df4f93-kserve-5d64f885b9-4gqgg"] Aug 04 16:43:47.572193 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.572133 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" podUID="db1e8884-9f29-4658-8227-e43e6fab03c7" containerName="main" containerID="cri-o://a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c" gracePeriod=45 Aug 04 16:43:47.825554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.825527 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:43:47.888523 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888489 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-tmp-dir\") pod \"db1e8884-9f29-4658-8227-e43e6fab03c7\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " Aug 04 16:43:47.888704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888550 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1e8884-9f29-4658-8227-e43e6fab03c7-tls-certs\") pod \"db1e8884-9f29-4658-8227-e43e6fab03c7\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " Aug 04 16:43:47.888704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888600 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr4t7\" (UniqueName: \"kubernetes.io/projected/db1e8884-9f29-4658-8227-e43e6fab03c7-kube-api-access-tr4t7\") pod \"db1e8884-9f29-4658-8227-e43e6fab03c7\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " Aug 04 16:43:47.888704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888697 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-home\") pod \"db1e8884-9f29-4658-8227-e43e6fab03c7\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " Aug 04 16:43:47.888864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888723 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-model-cache\") pod \"db1e8884-9f29-4658-8227-e43e6fab03c7\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " Aug 04 16:43:47.888864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888808 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db1e8884-9f29-4658-8227-e43e6fab03c7" (UID: "db1e8884-9f29-4658-8227-e43e6fab03c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:47.888977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888856 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-dshm\") pod \"db1e8884-9f29-4658-8227-e43e6fab03c7\" (UID: \"db1e8884-9f29-4658-8227-e43e6fab03c7\") " Aug 04 16:43:47.889033 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.888963 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "db1e8884-9f29-4658-8227-e43e6fab03c7" (UID: "db1e8884-9f29-4658-8227-e43e6fab03c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:47.889033 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.889003 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-home" (OuterVolumeSpecName: "home") pod "db1e8884-9f29-4658-8227-e43e6fab03c7" (UID: "db1e8884-9f29-4658-8227-e43e6fab03c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:47.889275 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.889256 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.889351 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.889279 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.889351 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.889295 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.890966 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.890942 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1e8884-9f29-4658-8227-e43e6fab03c7-kube-api-access-tr4t7" (OuterVolumeSpecName: "kube-api-access-tr4t7") pod "db1e8884-9f29-4658-8227-e43e6fab03c7" (UID: "db1e8884-9f29-4658-8227-e43e6fab03c7"). InnerVolumeSpecName "kube-api-access-tr4t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:43:47.891074 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.890961 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-dshm" (OuterVolumeSpecName: "dshm") pod "db1e8884-9f29-4658-8227-e43e6fab03c7" (UID: "db1e8884-9f29-4658-8227-e43e6fab03c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:43:47.891074 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.891031 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db1e8884-9f29-4658-8227-e43e6fab03c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db1e8884-9f29-4658-8227-e43e6fab03c7" (UID: "db1e8884-9f29-4658-8227-e43e6fab03c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:43:47.990460 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.990368 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db1e8884-9f29-4658-8227-e43e6fab03c7-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.990460 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.990399 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db1e8884-9f29-4658-8227-e43e6fab03c7-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:47.990460 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:47.990411 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr4t7\" (UniqueName: \"kubernetes.io/projected/db1e8884-9f29-4658-8227-e43e6fab03c7-kube-api-access-tr4t7\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:48.470932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.470893 2571 generic.go:356] "Generic (PLEG): container finished" podID="db1e8884-9f29-4658-8227-e43e6fab03c7" containerID="a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c" exitCode=0 Aug 04 16:43:48.471092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.470939 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" Aug 04 16:43:48.471092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.470973 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" event={"ID":"db1e8884-9f29-4658-8227-e43e6fab03c7","Type":"ContainerDied","Data":"a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c"} Aug 04 16:43:48.471092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.471019 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr" event={"ID":"db1e8884-9f29-4658-8227-e43e6fab03c7","Type":"ContainerDied","Data":"1ee77c919aeb7560c7f28ed83cbc7c276f05dac0517dae2c1816199731c36c26"} Aug 04 16:43:48.471092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.471037 2571 scope.go:117] "RemoveContainer" containerID="a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c" Aug 04 16:43:48.480926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.480905 2571 scope.go:117] "RemoveContainer" containerID="a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c" Aug 04 16:43:48.481225 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:43:48.481190 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c\": container with ID starting with a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c not found: ID does not exist" containerID="a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c" Aug 04 16:43:48.481336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.481230 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c"} err="failed to get container status \"a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c\": rpc error: code = NotFound desc = could not find container \"a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c\": container with ID starting with a9abde414b87b4c447609e8bc89903660e80b4e87ca47a70fa3e91680f9d8b4c not found: ID does not exist" Aug 04 16:43:48.493269 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.493189 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr"] Aug 04 16:43:48.495907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:48.495884 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1f0f0c/test-model-1f0f0c-kserve-6cfbb984cc-972tr"] Aug 04 16:43:49.364215 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:49.364165 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db1e8884-9f29-4658-8227-e43e6fab03c7" path="/var/lib/kubelet/pods/db1e8884-9f29-4658-8227-e43e6fab03c7/volumes" Aug 04 16:43:49.364692 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:49.364674 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" path="/var/lib/kubelet/pods/ea9f8d3b-084f-467d-99ce-f59ecc4bb61b/volumes" Aug 04 16:43:49.835958 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:49.835920 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:49.975073 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:49.975038 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:50.072644 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.072610 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:50.398311 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.398268 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:43:50.437023 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.436980 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr"] Aug 04 16:43:50.437449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437435 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" containerName="main" Aug 04 16:43:50.437504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437452 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" containerName="main" Aug 04 16:43:50.437504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437461 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ecf361-190b-4028-80c8-00097d16d1e5" containerName="revoke-keys" Aug 04 16:43:50.437504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437466 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ecf361-190b-4028-80c8-00097d16d1e5" containerName="revoke-keys" Aug 04 16:43:50.437504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437478 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1e8884-9f29-4658-8227-e43e6fab03c7" containerName="main" Aug 04 16:43:50.437504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437484 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1e8884-9f29-4658-8227-e43e6fab03c7" containerName="main" Aug 04 16:43:50.437654 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437539 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea9f8d3b-084f-467d-99ce-f59ecc4bb61b" containerName="main" Aug 04 16:43:50.437654 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437547 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ecf361-190b-4028-80c8-00097d16d1e5" containerName="revoke-keys" Aug 04 16:43:50.437654 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.437553 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1e8884-9f29-4658-8227-e43e6fab03c7" containerName="main" Aug 04 16:43:50.441649 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.441626 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" Aug 04 16:43:50.444594 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.444567 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:43:50.444749 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.444634 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:43:50.449356 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.449332 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr"] Aug 04 16:43:50.514971 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.514925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b271b3-f449-4088-a7f9-1d3619c2e1c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr\" (UID: \"f5b271b3-f449-4088-a7f9-1d3619c2e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" Aug 04 16:43:50.616285 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.616238 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b271b3-f449-4088-a7f9-1d3619c2e1c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr\" (UID: \"f5b271b3-f449-4088-a7f9-1d3619c2e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" Aug 04 16:43:50.616869 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.616849 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b271b3-f449-4088-a7f9-1d3619c2e1c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr\" (UID: \"f5b271b3-f449-4088-a7f9-1d3619c2e1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" Aug 04 16:43:50.755094 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.754993 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" Aug 04 16:43:50.903676 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:50.903565 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr"] Aug 04 16:43:50.907319 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:43:50.907290 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b271b3_f449_4088_a7f9_1d3619c2e1c6.slice/crio-3b11e0c07db8fabfa241761a43bb3cd86c9e965fc66f2da29f8beea350919ef5 WatchSource:0}: Error finding container 3b11e0c07db8fabfa241761a43bb3cd86c9e965fc66f2da29f8beea350919ef5: Status 404 returned error can't find the container with id 3b11e0c07db8fabfa241761a43bb3cd86c9e965fc66f2da29f8beea350919ef5 Aug 04 16:43:51.491677 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:51.491583 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" event={"ID":"f5b271b3-f449-4088-a7f9-1d3619c2e1c6","Type":"ContainerStarted","Data":"9892df2c8dcfbe770ee8b8c644c0af80b179970cca894385c211f1cfd3d0f8e0"} Aug 04 16:43:51.491677 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:51.491650 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" event={"ID":"f5b271b3-f449-4088-a7f9-1d3619c2e1c6","Type":"ContainerStarted","Data":"3b11e0c07db8fabfa241761a43bb3cd86c9e965fc66f2da29f8beea350919ef5"} Aug 04 16:43:51.511368 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:51.511320 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" podStartSLOduration=1.511304914 podStartE2EDuration="1.511304914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:43:51.509989126 +0000 UTC m=+2840.603589375" watchObservedRunningTime="2026-08-04 16:43:51.511304914 +0000 UTC m=+2840.604905162" Aug 04 16:43:52.498576 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:52.498538 2571 generic.go:356] "Generic (PLEG): container finished" podID="f5b271b3-f449-4088-a7f9-1d3619c2e1c6" containerID="9892df2c8dcfbe770ee8b8c644c0af80b179970cca894385c211f1cfd3d0f8e0" exitCode=0 Aug 04 16:43:52.498975 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:52.498619 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" event={"ID":"f5b271b3-f449-4088-a7f9-1d3619c2e1c6","Type":"ContainerDied","Data":"9892df2c8dcfbe770ee8b8c644c0af80b179970cca894385c211f1cfd3d0f8e0"} Aug 04 16:43:53.640036 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:53.640009 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" Aug 04 16:43:53.748712 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:53.748672 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b271b3-f449-4088-a7f9-1d3619c2e1c6-maas-api-service-ca\") pod \"f5b271b3-f449-4088-a7f9-1d3619c2e1c6\" (UID: \"f5b271b3-f449-4088-a7f9-1d3619c2e1c6\") " Aug 04 16:43:53.749061 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:53.749033 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b271b3-f449-4088-a7f9-1d3619c2e1c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5b271b3-f449-4088-a7f9-1d3619c2e1c6" (UID: "f5b271b3-f449-4088-a7f9-1d3619c2e1c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:43:53.849908 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:53.849871 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5b271b3-f449-4088-a7f9-1d3619c2e1c6-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:43:54.510655 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:54.510627 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" Aug 04 16:43:54.510833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:54.510666 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr" event={"ID":"f5b271b3-f449-4088-a7f9-1d3619c2e1c6","Type":"ContainerDied","Data":"3b11e0c07db8fabfa241761a43bb3cd86c9e965fc66f2da29f8beea350919ef5"} Aug 04 16:43:54.510833 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:43:54.510713 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b11e0c07db8fabfa241761a43bb3cd86c9e965fc66f2da29f8beea350919ef5" Aug 04 16:44:01.006430 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.006388 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:01.082851 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.082800 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:01.149339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.149297 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:01.200146 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.200106 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr"] Aug 04 16:44:01.202520 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.202498 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-1f0f0c-5p6hr"] Aug 04 16:44:01.220580 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.220531 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:01.364092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.364053 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5b271b3-f449-4088-a7f9-1d3619c2e1c6" path="/var/lib/kubelet/pods/f5b271b3-f449-4088-a7f9-1d3619c2e1c6/volumes" Aug 04 16:44:01.606336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:01.606301 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:03.211106 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:03.211070 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:44:07.347707 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:07.347665 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:07.454899 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:07.454858 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:07.692369 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:07.692258 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:07.983510 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:07.983411 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:08.362715 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:08.362678 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:08.858715 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:08.858679 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:09.001379 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:09.001333 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:11.507010 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:11.506971 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:11.614199 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:11.614160 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:12.787389 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:12.787347 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:13.000387 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:13.000349 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:13.411548 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:13.411505 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:13.500235 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:13.500173 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:15.087898 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:15.087850 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:44:36.583219 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:36.583171 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:38.844823 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:38.844787 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:40.778010 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:40.777969 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:40.869094 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:40.869054 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:42.488533 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:42.488435 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:42.964140 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:42.964104 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:43.120314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:43.120272 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:43.226704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:43.226614 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:48.326489 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:48.326449 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:44:55.502578 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:55.502538 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:44:58.553043 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.552998 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm"] Aug 04 16:44:58.553620 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.553599 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b271b3-f449-4088-a7f9-1d3619c2e1c6" containerName="revoke-keys" Aug 04 16:44:58.553701 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.553623 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b271b3-f449-4088-a7f9-1d3619c2e1c6" containerName="revoke-keys" Aug 04 16:44:58.553755 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.553720 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b271b3-f449-4088-a7f9-1d3619c2e1c6" containerName="revoke-keys" Aug 04 16:44:58.556955 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.556933 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" Aug 04 16:44:58.560288 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.560266 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:44:58.560416 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.560311 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:44:58.564571 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.564547 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm"] Aug 04 16:44:58.754958 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.754921 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcebe969-4620-4945-82d7-9f10308a14ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78e391-t7krm\" (UID: \"dcebe969-4620-4945-82d7-9f10308a14ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" Aug 04 16:44:58.856015 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.855974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcebe969-4620-4945-82d7-9f10308a14ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78e391-t7krm\" (UID: \"dcebe969-4620-4945-82d7-9f10308a14ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" Aug 04 16:44:58.856675 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.856652 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcebe969-4620-4945-82d7-9f10308a14ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78e391-t7krm\" (UID: \"dcebe969-4620-4945-82d7-9f10308a14ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" Aug 04 16:44:58.868953 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:58.868920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" Aug 04 16:44:59.216938 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:59.216902 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm"] Aug 04 16:44:59.218663 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:44:59.218629 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcebe969_4620_4945_82d7_9f10308a14ca.slice/crio-b930e2e881e7c00e4d8ab6057fc05cf528b3230829349e299653356a5e42ff7b WatchSource:0}: Error finding container b930e2e881e7c00e4d8ab6057fc05cf528b3230829349e299653356a5e42ff7b: Status 404 returned error can't find the container with id b930e2e881e7c00e4d8ab6057fc05cf528b3230829349e299653356a5e42ff7b Aug 04 16:44:59.887043 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:59.887001 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" event={"ID":"dcebe969-4620-4945-82d7-9f10308a14ca","Type":"ContainerStarted","Data":"2ddc5284b6f9487e5ab1c575cb78cf9b150eda6234d67f944817db89715b7844"} Aug 04 16:44:59.887650 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:59.887098 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" event={"ID":"dcebe969-4620-4945-82d7-9f10308a14ca","Type":"ContainerStarted","Data":"b930e2e881e7c00e4d8ab6057fc05cf528b3230829349e299653356a5e42ff7b"} Aug 04 16:44:59.905318 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:44:59.905262 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" podStartSLOduration=1.905242788 podStartE2EDuration="1.905242788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:44:59.902326883 +0000 UTC m=+2908.995927132" watchObservedRunningTime="2026-08-04 16:44:59.905242788 +0000 UTC m=+2908.998843037" Aug 04 16:45:00.133197 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.133151 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt"] Aug 04 16:45:00.136967 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.136938 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7"] Aug 04 16:45:00.137155 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.137057 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" Aug 04 16:45:00.140752 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.140730 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8"] Aug 04 16:45:00.140870 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.140851 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7" Aug 04 16:45:00.144589 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.144569 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" Aug 04 16:45:00.146777 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.146750 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt"] Aug 04 16:45:00.152102 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.152045 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8"] Aug 04 16:45:00.152102 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.152098 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" Aug 04 16:45:00.156167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.155401 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7"] Aug 04 16:45:00.162882 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.162859 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7" Aug 04 16:45:00.201547 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.201521 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" Aug 04 16:45:00.402569 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.402452 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8"] Aug 04 16:45:00.404701 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:45:00.404670 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70c9f3a_fc65_4d7b_a875_088e0eecd8f5.slice/crio-9eacb1d738b7077bbfc6de04038b7f566463d324376161bcd1a6d4c0a926b86d WatchSource:0}: Error finding container 9eacb1d738b7077bbfc6de04038b7f566463d324376161bcd1a6d4c0a926b86d: Status 404 returned error can't find the container with id 9eacb1d738b7077bbfc6de04038b7f566463d324376161bcd1a6d4c0a926b86d Aug 04 16:45:00.547445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.547409 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt"] Aug 04 16:45:00.548946 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:45:00.548915 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf21625c_f73d_484b_b029_e36b0a1849ea.slice/crio-64d5bd9fcd15af78a009ff7e6b67059f6cc02fcec5a82c62177ba00367b45606 WatchSource:0}: Error finding container 64d5bd9fcd15af78a009ff7e6b67059f6cc02fcec5a82c62177ba00367b45606: Status 404 returned error can't find the container with id 64d5bd9fcd15af78a009ff7e6b67059f6cc02fcec5a82c62177ba00367b45606 Aug 04 16:45:00.568265 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.568236 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7"] Aug 04 16:45:00.569763 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:45:00.569737 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ca3fa0_5997_4eba_9eb3_548422bc9bf1.slice/crio-d31ae8a21a207ac3e677739c4d6416b0228495a15e6aea6d7d06d4eb80cb5f29 WatchSource:0}: Error finding container d31ae8a21a207ac3e677739c4d6416b0228495a15e6aea6d7d06d4eb80cb5f29: Status 404 returned error can't find the container with id d31ae8a21a207ac3e677739c4d6416b0228495a15e6aea6d7d06d4eb80cb5f29 Aug 04 16:45:00.894704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.894672 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" event={"ID":"af21625c-f73d-484b-b029-e36b0a1849ea","Type":"ContainerStarted","Data":"964d86d8f697025af3f8e16ef510d0603161a945a213bc9950fc34076d1c329c"} Aug 04 16:45:00.895412 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.894726 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" event={"ID":"af21625c-f73d-484b-b029-e36b0a1849ea","Type":"ContainerStarted","Data":"64d5bd9fcd15af78a009ff7e6b67059f6cc02fcec5a82c62177ba00367b45606"} Aug 04 16:45:00.896305 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.896270 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" event={"ID":"f70c9f3a-fc65-4d7b-a875-088e0eecd8f5","Type":"ContainerStarted","Data":"d4de636b5aa8ae0842dc223e1ce137d2a665dd329d8c8c8cfe919230acc9ee15"} Aug 04 16:45:00.896440 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.896341 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" event={"ID":"f70c9f3a-fc65-4d7b-a875-088e0eecd8f5","Type":"ContainerStarted","Data":"9eacb1d738b7077bbfc6de04038b7f566463d324376161bcd1a6d4c0a926b86d"} Aug 04 16:45:00.897532 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.897506 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7" event={"ID":"b4ca3fa0-5997-4eba-9eb3-548422bc9bf1","Type":"ContainerStarted","Data":"d31ae8a21a207ac3e677739c4d6416b0228495a15e6aea6d7d06d4eb80cb5f29"} Aug 04 16:45:00.898988 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.898967 2571 generic.go:356] "Generic (PLEG): container finished" podID="dcebe969-4620-4945-82d7-9f10308a14ca" containerID="2ddc5284b6f9487e5ab1c575cb78cf9b150eda6234d67f944817db89715b7844" exitCode=0 Aug 04 16:45:00.899095 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.899007 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" event={"ID":"dcebe969-4620-4945-82d7-9f10308a14ca","Type":"ContainerDied","Data":"2ddc5284b6f9487e5ab1c575cb78cf9b150eda6234d67f944817db89715b7844"} Aug 04 16:45:00.912312 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.912265 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" podStartSLOduration=0.912251297 podStartE2EDuration="912.251297ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:45:00.91110607 +0000 UTC m=+2910.004706319" watchObservedRunningTime="2026-08-04 16:45:00.912251297 +0000 UTC m=+2910.005851544" Aug 04 16:45:00.943423 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:00.943376 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" podStartSLOduration=0.943358544 podStartE2EDuration="943.358544ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:45:00.941175951 +0000 UTC m=+2910.034776190" watchObservedRunningTime="2026-08-04 16:45:00.943358544 +0000 UTC m=+2910.036958770" Aug 04 16:45:01.906170 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:01.906134 2571 generic.go:356] "Generic (PLEG): container finished" podID="af21625c-f73d-484b-b029-e36b0a1849ea" containerID="964d86d8f697025af3f8e16ef510d0603161a945a213bc9950fc34076d1c329c" exitCode=0 Aug 04 16:45:01.906679 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:01.906235 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" event={"ID":"af21625c-f73d-484b-b029-e36b0a1849ea","Type":"ContainerDied","Data":"964d86d8f697025af3f8e16ef510d0603161a945a213bc9950fc34076d1c329c"} Aug 04 16:45:01.907984 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:01.907958 2571 generic.go:356] "Generic (PLEG): container finished" podID="f70c9f3a-fc65-4d7b-a875-088e0eecd8f5" containerID="d4de636b5aa8ae0842dc223e1ce137d2a665dd329d8c8c8cfe919230acc9ee15" exitCode=0 Aug 04 16:45:01.908119 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:01.908041 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" event={"ID":"f70c9f3a-fc65-4d7b-a875-088e0eecd8f5","Type":"ContainerDied","Data":"d4de636b5aa8ae0842dc223e1ce137d2a665dd329d8c8c8cfe919230acc9ee15"} Aug 04 16:45:01.909700 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:01.909678 2571 generic.go:356] "Generic (PLEG): container finished" podID="b4ca3fa0-5997-4eba-9eb3-548422bc9bf1" containerID="f995462e8d7eb7f3bdc9e953e77bf36d405377c21bc3c92b3c2dae8ec73613c2" exitCode=0 Aug 04 16:45:01.909820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:01.909760 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7" event={"ID":"b4ca3fa0-5997-4eba-9eb3-548422bc9bf1","Type":"ContainerDied","Data":"f995462e8d7eb7f3bdc9e953e77bf36d405377c21bc3c92b3c2dae8ec73613c2"} Aug 04 16:45:02.037558 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:02.037533 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" Aug 04 16:45:02.087932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:02.087897 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcebe969-4620-4945-82d7-9f10308a14ca-maas-api-service-ca\") pod \"dcebe969-4620-4945-82d7-9f10308a14ca\" (UID: \"dcebe969-4620-4945-82d7-9f10308a14ca\") " Aug 04 16:45:02.088309 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:02.088285 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcebe969-4620-4945-82d7-9f10308a14ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcebe969-4620-4945-82d7-9f10308a14ca" (UID: "dcebe969-4620-4945-82d7-9f10308a14ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:45:02.189128 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:02.189043 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcebe969-4620-4945-82d7-9f10308a14ca-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:45:02.923553 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:02.923516 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" Aug 04 16:45:02.923553 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:02.923527 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm" event={"ID":"dcebe969-4620-4945-82d7-9f10308a14ca","Type":"ContainerDied","Data":"b930e2e881e7c00e4d8ab6057fc05cf528b3230829349e299653356a5e42ff7b"} Aug 04 16:45:02.924059 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:02.923601 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b930e2e881e7c00e4d8ab6057fc05cf528b3230829349e299653356a5e42ff7b" Aug 04 16:45:03.068742 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.068711 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7" Aug 04 16:45:03.112667 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.112639 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" Aug 04 16:45:03.135153 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.135124 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" Aug 04 16:45:03.931076 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.931044 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7" Aug 04 16:45:03.931582 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.931084 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7" event={"ID":"b4ca3fa0-5997-4eba-9eb3-548422bc9bf1","Type":"ContainerDied","Data":"d31ae8a21a207ac3e677739c4d6416b0228495a15e6aea6d7d06d4eb80cb5f29"} Aug 04 16:45:03.931582 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.931134 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d31ae8a21a207ac3e677739c4d6416b0228495a15e6aea6d7d06d4eb80cb5f29" Aug 04 16:45:03.932496 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.932475 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" Aug 04 16:45:03.932653 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.932474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764365-hp6gt" event={"ID":"af21625c-f73d-484b-b029-e36b0a1849ea","Type":"ContainerDied","Data":"64d5bd9fcd15af78a009ff7e6b67059f6cc02fcec5a82c62177ba00367b45606"} Aug 04 16:45:03.932653 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.932622 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64d5bd9fcd15af78a009ff7e6b67059f6cc02fcec5a82c62177ba00367b45606" Aug 04 16:45:03.933922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.933900 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" event={"ID":"f70c9f3a-fc65-4d7b-a875-088e0eecd8f5","Type":"ContainerDied","Data":"9eacb1d738b7077bbfc6de04038b7f566463d324376161bcd1a6d4c0a926b86d"} Aug 04 16:45:03.934008 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.933933 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8" Aug 04 16:45:03.934066 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:03.933936 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eacb1d738b7077bbfc6de04038b7f566463d324376161bcd1a6d4c0a926b86d" Aug 04 16:45:11.688691 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.688639 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:11.697430 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.697399 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7"] Aug 04 16:45:11.700836 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.700808 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-78e391-29764365-wdts7"] Aug 04 16:45:11.777830 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.777788 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:11.844340 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.844292 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:11.904930 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.904889 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm"] Aug 04 16:45:11.907721 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.907695 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78e391-t7krm"] Aug 04 16:45:11.920785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:11.920748 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:12.306889 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:12.306850 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:13.364093 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:13.364058 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4ca3fa0-5997-4eba-9eb3-548422bc9bf1" path="/var/lib/kubelet/pods/b4ca3fa0-5997-4eba-9eb3-548422bc9bf1/volumes" Aug 04 16:45:13.364512 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:13.364465 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcebe969-4620-4945-82d7-9f10308a14ca" path="/var/lib/kubelet/pods/dcebe969-4620-4945-82d7-9f10308a14ca/volumes" Aug 04 16:45:14.182960 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:14.182920 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:45:18.404652 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:18.404621 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:18.504933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:18.504900 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:18.677435 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:18.677352 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:18.922006 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:18.921953 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:19.193336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193302 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5"] Aug 04 16:45:19.193689 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193676 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af21625c-f73d-484b-b029-e36b0a1849ea" containerName="cleanup" Aug 04 16:45:19.193754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193691 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="af21625c-f73d-484b-b029-e36b0a1849ea" containerName="cleanup" Aug 04 16:45:19.193754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193708 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4ca3fa0-5997-4eba-9eb3-548422bc9bf1" containerName="cleanup" Aug 04 16:45:19.193754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193713 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4ca3fa0-5997-4eba-9eb3-548422bc9bf1" containerName="cleanup" Aug 04 16:45:19.193754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193725 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcebe969-4620-4945-82d7-9f10308a14ca" containerName="revoke-keys" Aug 04 16:45:19.193754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193731 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcebe969-4620-4945-82d7-9f10308a14ca" containerName="revoke-keys" Aug 04 16:45:19.193754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193738 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70c9f3a-fc65-4d7b-a875-088e0eecd8f5" containerName="cleanup" Aug 04 16:45:19.193754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193744 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70c9f3a-fc65-4d7b-a875-088e0eecd8f5" containerName="cleanup" Aug 04 16:45:19.193983 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193809 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="af21625c-f73d-484b-b029-e36b0a1849ea" containerName="cleanup" Aug 04 16:45:19.193983 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193819 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4ca3fa0-5997-4eba-9eb3-548422bc9bf1" containerName="cleanup" Aug 04 16:45:19.193983 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193825 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70c9f3a-fc65-4d7b-a875-088e0eecd8f5" containerName="cleanup" Aug 04 16:45:19.193983 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.193833 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcebe969-4620-4945-82d7-9f10308a14ca" containerName="revoke-keys" Aug 04 16:45:19.197278 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.197258 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" Aug 04 16:45:19.200738 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.200716 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:45:19.218423 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.218393 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:45:19.219534 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.219501 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5"] Aug 04 16:45:19.256304 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.256262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6747ce1d-a229-4861-932e-7745e766bf4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e13060-j8lt5\" (UID: \"6747ce1d-a229-4861-932e-7745e766bf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" Aug 04 16:45:19.357328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.357285 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6747ce1d-a229-4861-932e-7745e766bf4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e13060-j8lt5\" (UID: \"6747ce1d-a229-4861-932e-7745e766bf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" Aug 04 16:45:19.357932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.357911 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6747ce1d-a229-4861-932e-7745e766bf4f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e13060-j8lt5\" (UID: \"6747ce1d-a229-4861-932e-7745e766bf4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" Aug 04 16:45:19.507856 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.507761 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" Aug 04 16:45:19.649737 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:19.649702 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5"] Aug 04 16:45:19.651769 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:45:19.651740 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6747ce1d_a229_4861_932e_7745e766bf4f.slice/crio-af6af2cc05afdfb4515af13d0c84b75d2464d43d80be67ccbac49fb82e4c8bb3 WatchSource:0}: Error finding container af6af2cc05afdfb4515af13d0c84b75d2464d43d80be67ccbac49fb82e4c8bb3: Status 404 returned error can't find the container with id af6af2cc05afdfb4515af13d0c84b75d2464d43d80be67ccbac49fb82e4c8bb3 Aug 04 16:45:20.033698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:20.033662 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" event={"ID":"6747ce1d-a229-4861-932e-7745e766bf4f","Type":"ContainerStarted","Data":"d2c695d6dcd202f4a8667cea31f0e346c2b44b10647ae6c4e0cc451f9e2a1dbf"} Aug 04 16:45:20.033913 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:20.033713 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" event={"ID":"6747ce1d-a229-4861-932e-7745e766bf4f","Type":"ContainerStarted","Data":"af6af2cc05afdfb4515af13d0c84b75d2464d43d80be67ccbac49fb82e4c8bb3"} Aug 04 16:45:20.049361 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:20.049301 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" podStartSLOduration=1.049279706 podStartE2EDuration="1.049279706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:45:20.04919875 +0000 UTC m=+2929.142799000" watchObservedRunningTime="2026-08-04 16:45:20.049279706 +0000 UTC m=+2929.142879950" Aug 04 16:45:21.041278 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:21.041242 2571 generic.go:356] "Generic (PLEG): container finished" podID="6747ce1d-a229-4861-932e-7745e766bf4f" containerID="d2c695d6dcd202f4a8667cea31f0e346c2b44b10647ae6c4e0cc451f9e2a1dbf" exitCode=0 Aug 04 16:45:21.041672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:21.041286 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" event={"ID":"6747ce1d-a229-4861-932e-7745e766bf4f","Type":"ContainerDied","Data":"d2c695d6dcd202f4a8667cea31f0e346c2b44b10647ae6c4e0cc451f9e2a1dbf"} Aug 04 16:45:22.189894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:22.189866 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" Aug 04 16:45:22.283334 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:22.283294 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6747ce1d-a229-4861-932e-7745e766bf4f-maas-api-service-ca\") pod \"6747ce1d-a229-4861-932e-7745e766bf4f\" (UID: \"6747ce1d-a229-4861-932e-7745e766bf4f\") " Aug 04 16:45:22.283698 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:22.283673 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6747ce1d-a229-4861-932e-7745e766bf4f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6747ce1d-a229-4861-932e-7745e766bf4f" (UID: "6747ce1d-a229-4861-932e-7745e766bf4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:45:22.384286 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:22.384248 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6747ce1d-a229-4861-932e-7745e766bf4f-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:45:23.057359 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:23.057327 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" Aug 04 16:45:23.057359 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:23.057345 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5" event={"ID":"6747ce1d-a229-4861-932e-7745e766bf4f","Type":"ContainerDied","Data":"af6af2cc05afdfb4515af13d0c84b75d2464d43d80be67ccbac49fb82e4c8bb3"} Aug 04 16:45:23.057579 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:23.057388 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af6af2cc05afdfb4515af13d0c84b75d2464d43d80be67ccbac49fb82e4c8bb3" Aug 04 16:45:24.199408 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:24.199365 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:24.300829 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:24.300794 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:30.384409 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.384367 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:30.385137 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.385112 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8"] Aug 04 16:45:30.387517 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.387496 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e13060-29764365-hdjf8"] Aug 04 16:45:30.479796 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.479761 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:30.540411 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.540370 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:30.578607 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.578570 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5"] Aug 04 16:45:30.580772 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.580751 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e13060-j8lt5"] Aug 04 16:45:30.597040 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.597000 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:30.989667 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:30.989613 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:31.363992 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:31.363953 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6747ce1d-a229-4861-932e-7745e766bf4f" path="/var/lib/kubelet/pods/6747ce1d-a229-4861-932e-7745e766bf4f/volumes" Aug 04 16:45:31.364378 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:31.364357 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70c9f3a-fc65-4d7b-a875-088e0eecd8f5" path="/var/lib/kubelet/pods/f70c9f3a-fc65-4d7b-a875-088e0eecd8f5/volumes" Aug 04 16:45:32.382376 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:32.382322 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:45:36.732999 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:36.732956 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:36.833514 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:36.833473 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:37.035765 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:37.035651 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:37.257186 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:37.257151 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:37.615966 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:37.615925 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:37.947153 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:37.947055 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:38.128871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:38.128833 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:41.030394 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:41.030344 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:41.126709 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:41.126677 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:42.154319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:42.154283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:42.260516 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:42.260479 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:42.647252 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:42.647213 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:42.748333 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:42.748285 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:45:45.191392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:45:45.191356 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:46:06.027583 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:06.027498 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:06.369693 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:06.369654 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:06.693926 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:06.693828 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:06.792771 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:06.792732 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:09.192669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:09.192626 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:09.294902 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:09.294854 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:10.529279 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:10.529239 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:10.654112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:10.654073 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:10.767217 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:10.767175 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:11.041217 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:11.041178 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:11.130886 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:11.130829 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:13.493239 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:13.492519 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:46:25.266903 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:25.266859 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:34.227930 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:34.227890 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:52.078643 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.078609 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2"] Aug 04 16:46:52.079091 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.079015 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6747ce1d-a229-4861-932e-7745e766bf4f" containerName="revoke-keys" Aug 04 16:46:52.079091 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.079026 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6747ce1d-a229-4861-932e-7745e766bf4f" containerName="revoke-keys" Aug 04 16:46:52.079164 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.079110 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6747ce1d-a229-4861-932e-7745e766bf4f" containerName="revoke-keys" Aug 04 16:46:52.082265 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.082249 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.084828 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.084809 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"openshift-service-ca.crt\"" Aug 04 16:46:52.085916 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.085896 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"default-dockercfg-bgnk2\"" Aug 04 16:46:52.085916 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.085905 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"kube-root-ca.crt\"" Aug 04 16:46:52.086252 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.086237 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"auth-test-model-90d707-kserve-self-signed-certs\"" Aug 04 16:46:52.091565 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.091542 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2"] Aug 04 16:46:52.096217 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.096176 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:46:52.222484 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.222445 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d418d-069f-4cbc-9457-5451a2f9cc35-tls-certs\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.222688 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.222571 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-dshm\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.222688 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.222615 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-model-cache\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.222816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.222729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-home\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.222816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.222768 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-tmp-dir\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.222816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.222805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8pq\" (UniqueName: \"kubernetes.io/projected/a55d418d-069f-4cbc-9457-5451a2f9cc35-kube-api-access-jw8pq\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.323982 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.323947 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-tmp-dir\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.323982 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.323992 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8pq\" (UniqueName: \"kubernetes.io/projected/a55d418d-069f-4cbc-9457-5451a2f9cc35-kube-api-access-jw8pq\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.324316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.324033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d418d-069f-4cbc-9457-5451a2f9cc35-tls-certs\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.324316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.324089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-dshm\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.324316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.324122 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-model-cache\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.324316 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.324176 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-home\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.324563 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.324493 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-tmp-dir\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.324563 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.324533 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-model-cache\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.324636 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.324617 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-home\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.326481 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.326453 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-dshm\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.326706 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.326688 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d418d-069f-4cbc-9457-5451a2f9cc35-tls-certs\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.332308 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.332248 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8pq\" (UniqueName: \"kubernetes.io/projected/a55d418d-069f-4cbc-9457-5451a2f9cc35-kube-api-access-jw8pq\") pod \"auth-test-model-90d707-kserve-55d7756495-x74g2\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.393192 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.393159 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:46:52.534670 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.534637 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2"] Aug 04 16:46:52.536102 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:46:52.536077 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55d418d_069f_4cbc_9457_5451a2f9cc35.slice/crio-735e23a5a860639040fc5e0fcba4ca61e6a07d927654d564cf1f167602225112 WatchSource:0}: Error finding container 735e23a5a860639040fc5e0fcba4ca61e6a07d927654d564cf1f167602225112: Status 404 returned error can't find the container with id 735e23a5a860639040fc5e0fcba4ca61e6a07d927654d564cf1f167602225112 Aug 04 16:46:52.586393 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:52.586316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" event={"ID":"a55d418d-069f-4cbc-9457-5451a2f9cc35","Type":"ContainerStarted","Data":"735e23a5a860639040fc5e0fcba4ca61e6a07d927654d564cf1f167602225112"} Aug 04 16:46:53.594584 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:53.594543 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" event={"ID":"a55d418d-069f-4cbc-9457-5451a2f9cc35","Type":"ContainerStarted","Data":"19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93"} Aug 04 16:46:53.614765 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:53.614709 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" podStartSLOduration=1.614693317 podStartE2EDuration="1.614693317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:46:53.612437809 +0000 UTC m=+3022.706038058" watchObservedRunningTime="2026-08-04 16:46:53.614693317 +0000 UTC m=+3022.708293565" Aug 04 16:46:54.982654 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:46:54.982618 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:47:02.394151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:02.394112 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:47:02.394660 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:02.394163 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:47:02.406704 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:02.406679 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:47:02.659298 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:02.659197 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:47:11.068662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.068628 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v"] Aug 04 16:47:11.072568 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.072549 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.077089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.077053 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"auth-test-model-5a3cb8-kserve-self-signed-certs\"" Aug 04 16:47:11.077089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.077071 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"kube-root-ca.crt\"" Aug 04 16:47:11.077315 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.077095 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"default-dockercfg-xcqvj\"" Aug 04 16:47:11.077442 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.077421 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"openshift-service-ca.crt\"" Aug 04 16:47:11.082722 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.082701 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v"] Aug 04 16:47:11.093031 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.092994 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:47:11.200372 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.200334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-home\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.200372 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.200373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ee2241-1ccd-43b4-87c4-009e98cb712f-tls-certs\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.200599 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.200455 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br5n7\" (UniqueName: \"kubernetes.io/projected/06ee2241-1ccd-43b4-87c4-009e98cb712f-kube-api-access-br5n7\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.200599 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.200546 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-tmp-dir\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.200669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.200604 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-dshm\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.200669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.200654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-model-cache\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301115 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301078 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-model-cache\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301115 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301121 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-home\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301139 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ee2241-1ccd-43b4-87c4-009e98cb712f-tls-certs\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301172 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br5n7\" (UniqueName: \"kubernetes.io/projected/06ee2241-1ccd-43b4-87c4-009e98cb712f-kube-api-access-br5n7\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301228 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-tmp-dir\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301392 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301268 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-dshm\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301588 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301568 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-model-cache\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301633 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301600 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-tmp-dir\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.301766 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.301750 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-home\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.303453 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.303429 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-dshm\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.303922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.303900 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ee2241-1ccd-43b4-87c4-009e98cb712f-tls-certs\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.310008 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.309983 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br5n7\" (UniqueName: \"kubernetes.io/projected/06ee2241-1ccd-43b4-87c4-009e98cb712f-kube-api-access-br5n7\") pod \"auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.383056 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.382965 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:11.517654 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.517622 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v"] Aug 04 16:47:11.520140 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:47:11.520109 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06ee2241_1ccd_43b4_87c4_009e98cb712f.slice/crio-d86a2f859d6d17d1be3bf89b4b7324f57954f2d162bf584a969e265196a35a97 WatchSource:0}: Error finding container d86a2f859d6d17d1be3bf89b4b7324f57954f2d162bf584a969e265196a35a97: Status 404 returned error can't find the container with id d86a2f859d6d17d1be3bf89b4b7324f57954f2d162bf584a969e265196a35a97 Aug 04 16:47:11.699713 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.699623 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" event={"ID":"06ee2241-1ccd-43b4-87c4-009e98cb712f","Type":"ContainerStarted","Data":"a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0"} Aug 04 16:47:11.699713 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.699669 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" event={"ID":"06ee2241-1ccd-43b4-87c4-009e98cb712f","Type":"ContainerStarted","Data":"d86a2f859d6d17d1be3bf89b4b7324f57954f2d162bf584a969e265196a35a97"} Aug 04 16:47:11.718590 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:11.718537 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" podStartSLOduration=0.718521581 podStartE2EDuration="718.521581ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:47:11.716891786 +0000 UTC m=+3040.810492026" watchObservedRunningTime="2026-08-04 16:47:11.718521581 +0000 UTC m=+3040.812121828" Aug 04 16:47:13.777652 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:13.777614 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:47:21.383125 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:21.383089 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:21.383553 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:21.383140 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:21.395840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:21.395819 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:21.778007 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:21.777922 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:47:47.336776 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:47.336734 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:47:47.386950 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:47.386916 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:47:49.982074 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:47:49.982042 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:48:03.580301 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:48:03.580263 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:48:13.998530 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:48:13.998491 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:48:29.980541 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:48:29.980506 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:48:43.376468 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:48:43.376428 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:48:51.678824 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:48:51.678792 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:48:59.988015 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:48:59.987921 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:49:07.034592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:49:07.034556 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:49:15.175692 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:49:15.175655 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:49:25.986053 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:49:25.986018 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:49:41.283795 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:49:41.283756 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:49:54.985785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:49:54.985743 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:50:01.779620 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:01.779586 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:50:11.686884 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:11.686842 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:50:22.381077 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:22.381043 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:50:31.888355 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:31.888268 2571 scope.go:117] "RemoveContainer" containerID="27be5f306e4e65c60e0815b6d495213491e6f8cf89e8f10e946bf52d8ca621db" Aug 04 16:50:31.897228 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:31.897189 2571 scope.go:117] "RemoveContainer" containerID="9892df2c8dcfbe770ee8b8c644c0af80b179970cca894385c211f1cfd3d0f8e0" Aug 04 16:50:33.483290 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:33.483260 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:50:48.381659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:48.381628 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:50:52.358585 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:52.358541 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2"] Aug 04 16:50:52.673728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:52.673643 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:50:53.269825 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:53.269793 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v"] Aug 04 16:50:54.132059 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:54.132015 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:50:57.715070 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:57.715031 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:50:58.456344 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:58.456281 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:50:59.685804 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:50:59.685768 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:51:07.378487 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.378444 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" podUID="a55d418d-069f-4cbc-9457-5451a2f9cc35" containerName="main" containerID="cri-o://19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93" gracePeriod=45 Aug 04 16:51:07.628472 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.628449 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:51:07.778648 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778572 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-model-cache\") pod \"a55d418d-069f-4cbc-9457-5451a2f9cc35\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " Aug 04 16:51:07.778785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778654 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw8pq\" (UniqueName: \"kubernetes.io/projected/a55d418d-069f-4cbc-9457-5451a2f9cc35-kube-api-access-jw8pq\") pod \"a55d418d-069f-4cbc-9457-5451a2f9cc35\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " Aug 04 16:51:07.778785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778686 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d418d-069f-4cbc-9457-5451a2f9cc35-tls-certs\") pod \"a55d418d-069f-4cbc-9457-5451a2f9cc35\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " Aug 04 16:51:07.778785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778772 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-dshm\") pod \"a55d418d-069f-4cbc-9457-5451a2f9cc35\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " Aug 04 16:51:07.778941 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778795 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-home\") pod \"a55d418d-069f-4cbc-9457-5451a2f9cc35\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " Aug 04 16:51:07.778941 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778871 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-model-cache" (OuterVolumeSpecName: "model-cache") pod "a55d418d-069f-4cbc-9457-5451a2f9cc35" (UID: "a55d418d-069f-4cbc-9457-5451a2f9cc35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:07.779057 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778934 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-tmp-dir\") pod \"a55d418d-069f-4cbc-9457-5451a2f9cc35\" (UID: \"a55d418d-069f-4cbc-9457-5451a2f9cc35\") " Aug 04 16:51:07.779057 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.778993 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-home" (OuterVolumeSpecName: "home") pod "a55d418d-069f-4cbc-9457-5451a2f9cc35" (UID: "a55d418d-069f-4cbc-9457-5451a2f9cc35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:07.779171 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.779124 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a55d418d-069f-4cbc-9457-5451a2f9cc35" (UID: "a55d418d-069f-4cbc-9457-5451a2f9cc35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:07.779320 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.779302 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:07.779401 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.779320 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:07.779401 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.779330 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:07.780886 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.780862 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55d418d-069f-4cbc-9457-5451a2f9cc35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a55d418d-069f-4cbc-9457-5451a2f9cc35" (UID: "a55d418d-069f-4cbc-9457-5451a2f9cc35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:51:07.781005 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.780962 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-dshm" (OuterVolumeSpecName: "dshm") pod "a55d418d-069f-4cbc-9457-5451a2f9cc35" (UID: "a55d418d-069f-4cbc-9457-5451a2f9cc35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:07.781005 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.780961 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55d418d-069f-4cbc-9457-5451a2f9cc35-kube-api-access-jw8pq" (OuterVolumeSpecName: "kube-api-access-jw8pq") pod "a55d418d-069f-4cbc-9457-5451a2f9cc35" (UID: "a55d418d-069f-4cbc-9457-5451a2f9cc35"). InnerVolumeSpecName "kube-api-access-jw8pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:51:07.880366 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.880338 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55d418d-069f-4cbc-9457-5451a2f9cc35-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:07.880366 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.880361 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55d418d-069f-4cbc-9457-5451a2f9cc35-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:07.880366 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:07.880371 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8pq\" (UniqueName: \"kubernetes.io/projected/a55d418d-069f-4cbc-9457-5451a2f9cc35-kube-api-access-jw8pq\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:08.114963 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.114928 2571 generic.go:356] "Generic (PLEG): container finished" podID="a55d418d-069f-4cbc-9457-5451a2f9cc35" containerID="19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93" exitCode=0 Aug 04 16:51:08.115126 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.114967 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" Aug 04 16:51:08.115126 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.115014 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" event={"ID":"a55d418d-069f-4cbc-9457-5451a2f9cc35","Type":"ContainerDied","Data":"19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93"} Aug 04 16:51:08.115126 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.115061 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2" event={"ID":"a55d418d-069f-4cbc-9457-5451a2f9cc35","Type":"ContainerDied","Data":"735e23a5a860639040fc5e0fcba4ca61e6a07d927654d564cf1f167602225112"} Aug 04 16:51:08.115126 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.115080 2571 scope.go:117] "RemoveContainer" containerID="19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93" Aug 04 16:51:08.125048 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.125030 2571 scope.go:117] "RemoveContainer" containerID="19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93" Aug 04 16:51:08.125344 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:51:08.125325 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93\": container with ID starting with 19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93 not found: ID does not exist" containerID="19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93" Aug 04 16:51:08.125407 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.125352 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93"} err="failed to get container status \"19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93\": rpc error: code = NotFound desc = could not find container \"19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93\": container with ID starting with 19fa34d1f0ddf215bcc0c5b8cf8f91e455078b48447f3d25dbd47bc98eef1c93 not found: ID does not exist" Aug 04 16:51:08.137110 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.137089 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2"] Aug 04 16:51:08.140912 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.140892 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/auth-test-model-90d707-kserve-55d7756495-x74g2"] Aug 04 16:51:08.288521 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.288475 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" podUID="06ee2241-1ccd-43b4-87c4-009e98cb712f" containerName="main" containerID="cri-o://a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0" gracePeriod=45 Aug 04 16:51:08.530950 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.530926 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:51:08.686846 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.686768 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-model-cache\") pod \"06ee2241-1ccd-43b4-87c4-009e98cb712f\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " Aug 04 16:51:08.686846 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.686807 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-dshm\") pod \"06ee2241-1ccd-43b4-87c4-009e98cb712f\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " Aug 04 16:51:08.687079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.686852 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ee2241-1ccd-43b4-87c4-009e98cb712f-tls-certs\") pod \"06ee2241-1ccd-43b4-87c4-009e98cb712f\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " Aug 04 16:51:08.687079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.686877 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-home\") pod \"06ee2241-1ccd-43b4-87c4-009e98cb712f\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " Aug 04 16:51:08.687079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.686899 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-tmp-dir\") pod \"06ee2241-1ccd-43b4-87c4-009e98cb712f\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " Aug 04 16:51:08.687079 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.686946 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-br5n7\" (UniqueName: \"kubernetes.io/projected/06ee2241-1ccd-43b4-87c4-009e98cb712f-kube-api-access-br5n7\") pod \"06ee2241-1ccd-43b4-87c4-009e98cb712f\" (UID: \"06ee2241-1ccd-43b4-87c4-009e98cb712f\") " Aug 04 16:51:08.687307 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.687125 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-model-cache" (OuterVolumeSpecName: "model-cache") pod "06ee2241-1ccd-43b4-87c4-009e98cb712f" (UID: "06ee2241-1ccd-43b4-87c4-009e98cb712f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:08.687307 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.687169 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06ee2241-1ccd-43b4-87c4-009e98cb712f" (UID: "06ee2241-1ccd-43b4-87c4-009e98cb712f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:08.687307 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.687186 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-home" (OuterVolumeSpecName: "home") pod "06ee2241-1ccd-43b4-87c4-009e98cb712f" (UID: "06ee2241-1ccd-43b4-87c4-009e98cb712f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:08.687307 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.687278 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:08.689513 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.689488 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ee2241-1ccd-43b4-87c4-009e98cb712f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06ee2241-1ccd-43b4-87c4-009e98cb712f" (UID: "06ee2241-1ccd-43b4-87c4-009e98cb712f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:51:08.689639 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.689535 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-dshm" (OuterVolumeSpecName: "dshm") pod "06ee2241-1ccd-43b4-87c4-009e98cb712f" (UID: "06ee2241-1ccd-43b4-87c4-009e98cb712f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:51:08.689639 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.689571 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ee2241-1ccd-43b4-87c4-009e98cb712f-kube-api-access-br5n7" (OuterVolumeSpecName: "kube-api-access-br5n7") pod "06ee2241-1ccd-43b4-87c4-009e98cb712f" (UID: "06ee2241-1ccd-43b4-87c4-009e98cb712f"). InnerVolumeSpecName "kube-api-access-br5n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:51:08.788424 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.788395 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br5n7\" (UniqueName: \"kubernetes.io/projected/06ee2241-1ccd-43b4-87c4-009e98cb712f-kube-api-access-br5n7\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:08.788424 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.788421 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:08.788573 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.788430 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06ee2241-1ccd-43b4-87c4-009e98cb712f-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:08.788573 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.788439 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:08.788573 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:08.788447 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06ee2241-1ccd-43b4-87c4-009e98cb712f-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:51:09.122361 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.122325 2571 generic.go:356] "Generic (PLEG): container finished" podID="06ee2241-1ccd-43b4-87c4-009e98cb712f" containerID="a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0" exitCode=0 Aug 04 16:51:09.122549 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.122375 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" Aug 04 16:51:09.122549 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.122408 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" event={"ID":"06ee2241-1ccd-43b4-87c4-009e98cb712f","Type":"ContainerDied","Data":"a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0"} Aug 04 16:51:09.122549 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.122451 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v" event={"ID":"06ee2241-1ccd-43b4-87c4-009e98cb712f","Type":"ContainerDied","Data":"d86a2f859d6d17d1be3bf89b4b7324f57954f2d162bf584a969e265196a35a97"} Aug 04 16:51:09.122549 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.122474 2571 scope.go:117] "RemoveContainer" containerID="a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0" Aug 04 16:51:09.131969 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.131955 2571 scope.go:117] "RemoveContainer" containerID="a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0" Aug 04 16:51:09.132264 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:51:09.132244 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0\": container with ID starting with a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0 not found: ID does not exist" containerID="a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0" Aug 04 16:51:09.132347 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.132271 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0"} err="failed to get container status \"a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0\": rpc error: code = NotFound desc = could not find container \"a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0\": container with ID starting with a88b2599300a31e9fb7e9d058a24464f1945260086fa39de1115d3fb51679fe0 not found: ID does not exist" Aug 04 16:51:09.145750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.145730 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v"] Aug 04 16:51:09.149559 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.149537 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/auth-test-model-5a3cb8-kserve-6c7fb4bdc-mdb5v"] Aug 04 16:51:09.368736 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.368699 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06ee2241-1ccd-43b4-87c4-009e98cb712f" path="/var/lib/kubelet/pods/06ee2241-1ccd-43b4-87c4-009e98cb712f/volumes" Aug 04 16:51:09.369094 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:09.369081 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a55d418d-069f-4cbc-9457-5451a2f9cc35" path="/var/lib/kubelet/pods/a55d418d-069f-4cbc-9457-5451a2f9cc35/volumes" Aug 04 16:51:19.496122 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:19.496084 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:51:19.858319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:19.858282 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:51:19.952577 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:19.952550 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:51:31.922502 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:31.922468 2571 scope.go:117] "RemoveContainer" containerID="d4de636b5aa8ae0842dc223e1ce137d2a665dd329d8c8c8cfe919230acc9ee15" Aug 04 16:51:31.931279 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:31.931247 2571 scope.go:117] "RemoveContainer" containerID="2ddc5284b6f9487e5ab1c575cb78cf9b150eda6234d67f944817db89715b7844" Aug 04 16:51:31.938981 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:31.938962 2571 scope.go:117] "RemoveContainer" containerID="d2c695d6dcd202f4a8667cea31f0e346c2b44b10647ae6c4e0cc451f9e2a1dbf" Aug 04 16:51:31.946420 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:31.946403 2571 scope.go:117] "RemoveContainer" containerID="f995462e8d7eb7f3bdc9e953e77bf36d405377c21bc3c92b3c2dae8ec73613c2" Aug 04 16:51:38.833715 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:38.833672 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:51:38.934410 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:38.934377 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:51:39.330115 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.330078 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2"] Aug 04 16:51:39.330480 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.330466 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a55d418d-069f-4cbc-9457-5451a2f9cc35" containerName="main" Aug 04 16:51:39.330480 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.330480 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a55d418d-069f-4cbc-9457-5451a2f9cc35" containerName="main" Aug 04 16:51:39.330613 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.330501 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06ee2241-1ccd-43b4-87c4-009e98cb712f" containerName="main" Aug 04 16:51:39.330613 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.330507 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="06ee2241-1ccd-43b4-87c4-009e98cb712f" containerName="main" Aug 04 16:51:39.330613 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.330564 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="06ee2241-1ccd-43b4-87c4-009e98cb712f" containerName="main" Aug 04 16:51:39.330613 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.330575 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a55d418d-069f-4cbc-9457-5451a2f9cc35" containerName="main" Aug 04 16:51:39.335361 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.335329 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.337183 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.337153 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2"] Aug 04 16:51:39.339528 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.339310 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"openshift-service-ca.crt\"" Aug 04 16:51:39.339528 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.339348 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"kube-root-ca.crt\"" Aug 04 16:51:39.339528 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.339390 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"default-dockercfg-xcqvj\"" Aug 04 16:51:39.339528 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.339310 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"sub-test-model-5a3cb8-kserve-self-signed-certs\"" Aug 04 16:51:39.363250 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.363193 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:51:39.455016 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.454983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7232160-2148-4f0d-8383-b4c7726ff2d1-tls-certs\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.455170 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.455022 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-model-cache\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.455170 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.455055 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67vkz\" (UniqueName: \"kubernetes.io/projected/c7232160-2148-4f0d-8383-b4c7726ff2d1-kube-api-access-67vkz\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.455170 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.455136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-home\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.455296 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.455177 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-dshm\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.455296 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.455271 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-tmp-dir\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67vkz\" (UniqueName: \"kubernetes.io/projected/c7232160-2148-4f0d-8383-b4c7726ff2d1-kube-api-access-67vkz\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556381 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-home\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556381 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-dshm\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556381 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556346 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-tmp-dir\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556519 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556397 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7232160-2148-4f0d-8383-b4c7726ff2d1-tls-certs\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556519 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-model-cache\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556789 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556662 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-home\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556789 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556719 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-tmp-dir\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.556789 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.556747 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-model-cache\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.558667 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.558644 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-dshm\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.558947 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.558922 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7232160-2148-4f0d-8383-b4c7726ff2d1-tls-certs\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.566355 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.566329 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67vkz\" (UniqueName: \"kubernetes.io/projected/c7232160-2148-4f0d-8383-b4c7726ff2d1-kube-api-access-67vkz\") pod \"sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.648915 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.648849 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:39.787257 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:39.787227 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2"] Aug 04 16:51:39.789108 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:51:39.789085 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7232160_2148_4f0d_8383_b4c7726ff2d1.slice/crio-429e3e9f8424e118ed30552849564d9aa01a749c3ec7aeedc16c046dd86b06eb WatchSource:0}: Error finding container 429e3e9f8424e118ed30552849564d9aa01a749c3ec7aeedc16c046dd86b06eb: Status 404 returned error can't find the container with id 429e3e9f8424e118ed30552849564d9aa01a749c3ec7aeedc16c046dd86b06eb Aug 04 16:51:40.296350 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:40.296308 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" event={"ID":"c7232160-2148-4f0d-8383-b4c7726ff2d1","Type":"ContainerStarted","Data":"350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91"} Aug 04 16:51:40.296728 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:40.296360 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" event={"ID":"c7232160-2148-4f0d-8383-b4c7726ff2d1","Type":"ContainerStarted","Data":"429e3e9f8424e118ed30552849564d9aa01a749c3ec7aeedc16c046dd86b06eb"} Aug 04 16:51:40.315705 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:40.315661 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" podStartSLOduration=1.315648408 podStartE2EDuration="1.315648408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:51:40.314264309 +0000 UTC m=+3309.407864558" watchObservedRunningTime="2026-08-04 16:51:40.315648408 +0000 UTC m=+3309.409248656" Aug 04 16:51:41.984745 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:41.984712 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:51:49.649367 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:49.649330 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:49.649799 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:49.649390 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:49.661539 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:49.661518 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:51:50.367325 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:51:50.367300 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:52:21.575571 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:21.575538 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:52:33.889105 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:33.889066 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:52:45.298383 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:45.298340 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:52:50.664726 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:50.664691 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2"] Aug 04 16:52:51.242505 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:51.242461 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:52:51.305404 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:51.305371 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:52:56.774618 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:56.774570 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:52:57.667268 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.667226 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn"] Aug 04 16:52:57.670802 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.670781 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.674566 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.674534 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"default-dockercfg-bgnk2\"" Aug 04 16:52:57.674566 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.674547 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"rate-test-model-90d707-kserve-self-signed-certs\"" Aug 04 16:52:57.674748 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.674567 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"openshift-service-ca.crt\"" Aug 04 16:52:57.674748 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.674548 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90d707\"/\"kube-root-ca.crt\"" Aug 04 16:52:57.679432 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.679404 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn"] Aug 04 16:52:57.763904 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.763871 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdb9d\" (UniqueName: \"kubernetes.io/projected/0850def6-f349-4e9b-b92e-b9ee4274dfb2-kube-api-access-xdb9d\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.764089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.763936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tls-certs\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.764089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.763986 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tmp-dir\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.764089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.764010 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-home\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.764089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.764033 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-dshm\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.764330 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.764142 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-model-cache\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.778182 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.778155 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:52:57.864826 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.864793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-model-cache\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.864838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdb9d\" (UniqueName: \"kubernetes.io/projected/0850def6-f349-4e9b-b92e-b9ee4274dfb2-kube-api-access-xdb9d\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.864885 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tls-certs\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.864920 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tmp-dir\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865001 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.864945 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-home\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865249 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.865065 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-dshm\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865313 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.865268 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-model-cache\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865313 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.865281 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tmp-dir\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.865490 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.865461 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-home\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.867227 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.867172 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-dshm\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.867427 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.867412 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tls-certs\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.873565 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.873540 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdb9d\" (UniqueName: \"kubernetes.io/projected/0850def6-f349-4e9b-b92e-b9ee4274dfb2-kube-api-access-xdb9d\") pod \"rate-test-model-90d707-kserve-7c6587c45f-s5xvn\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:57.981942 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:57.981861 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:52:58.125735 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:58.125699 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn"] Aug 04 16:52:58.126802 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:52:58.126773 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0850def6_f349_4e9b_b92e_b9ee4274dfb2.slice/crio-9e655ccbc3f88494e22c0d3a38717747db524f77b885a67efae7af402491e010 WatchSource:0}: Error finding container 9e655ccbc3f88494e22c0d3a38717747db524f77b885a67efae7af402491e010: Status 404 returned error can't find the container with id 9e655ccbc3f88494e22c0d3a38717747db524f77b885a67efae7af402491e010 Aug 04 16:52:58.747215 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:58.747163 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" event={"ID":"0850def6-f349-4e9b-b92e-b9ee4274dfb2","Type":"ContainerStarted","Data":"1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f"} Aug 04 16:52:58.747388 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:58.747224 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" event={"ID":"0850def6-f349-4e9b-b92e-b9ee4274dfb2","Type":"ContainerStarted","Data":"9e655ccbc3f88494e22c0d3a38717747db524f77b885a67efae7af402491e010"} Aug 04 16:52:58.763988 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:52:58.763934 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" podStartSLOduration=1.763915852 podStartE2EDuration="1.763915852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:52:58.763534456 +0000 UTC m=+3387.857134705" watchObservedRunningTime="2026-08-04 16:52:58.763915852 +0000 UTC m=+3387.857516101" Aug 04 16:53:04.081877 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:04.081832 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:53:05.683615 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:05.683572 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" podUID="c7232160-2148-4f0d-8383-b4c7726ff2d1" containerName="main" containerID="cri-o://350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91" gracePeriod=45 Aug 04 16:53:05.939395 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:05.939325 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:53:06.040101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040067 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-tmp-dir\") pod \"c7232160-2148-4f0d-8383-b4c7726ff2d1\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " Aug 04 16:53:06.040101 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040107 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-home\") pod \"c7232160-2148-4f0d-8383-b4c7726ff2d1\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " Aug 04 16:53:06.040400 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040161 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-model-cache\") pod \"c7232160-2148-4f0d-8383-b4c7726ff2d1\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " Aug 04 16:53:06.040400 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040188 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7232160-2148-4f0d-8383-b4c7726ff2d1-tls-certs\") pod \"c7232160-2148-4f0d-8383-b4c7726ff2d1\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " Aug 04 16:53:06.040400 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040256 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67vkz\" (UniqueName: \"kubernetes.io/projected/c7232160-2148-4f0d-8383-b4c7726ff2d1-kube-api-access-67vkz\") pod \"c7232160-2148-4f0d-8383-b4c7726ff2d1\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " Aug 04 16:53:06.040400 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040291 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-dshm\") pod \"c7232160-2148-4f0d-8383-b4c7726ff2d1\" (UID: \"c7232160-2148-4f0d-8383-b4c7726ff2d1\") " Aug 04 16:53:06.040579 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040462 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7232160-2148-4f0d-8383-b4c7726ff2d1" (UID: "c7232160-2148-4f0d-8383-b4c7726ff2d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:06.040579 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040474 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7232160-2148-4f0d-8383-b4c7726ff2d1" (UID: "c7232160-2148-4f0d-8383-b4c7726ff2d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:06.040672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040653 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:53:06.040710 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040678 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:53:06.040871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.040848 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-home" (OuterVolumeSpecName: "home") pod "c7232160-2148-4f0d-8383-b4c7726ff2d1" (UID: "c7232160-2148-4f0d-8383-b4c7726ff2d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:06.042550 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.042519 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-dshm" (OuterVolumeSpecName: "dshm") pod "c7232160-2148-4f0d-8383-b4c7726ff2d1" (UID: "c7232160-2148-4f0d-8383-b4c7726ff2d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:53:06.042651 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.042523 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7232160-2148-4f0d-8383-b4c7726ff2d1-kube-api-access-67vkz" (OuterVolumeSpecName: "kube-api-access-67vkz") pod "c7232160-2148-4f0d-8383-b4c7726ff2d1" (UID: "c7232160-2148-4f0d-8383-b4c7726ff2d1"). InnerVolumeSpecName "kube-api-access-67vkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:53:06.042651 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.042569 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7232160-2148-4f0d-8383-b4c7726ff2d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7232160-2148-4f0d-8383-b4c7726ff2d1" (UID: "c7232160-2148-4f0d-8383-b4c7726ff2d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:53:06.141309 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.141274 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:53:06.141309 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.141304 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7232160-2148-4f0d-8383-b4c7726ff2d1-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:53:06.141309 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.141315 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67vkz\" (UniqueName: \"kubernetes.io/projected/c7232160-2148-4f0d-8383-b4c7726ff2d1-kube-api-access-67vkz\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:53:06.141541 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.141325 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7232160-2148-4f0d-8383-b4c7726ff2d1-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:53:06.790827 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.790794 2571 generic.go:356] "Generic (PLEG): container finished" podID="c7232160-2148-4f0d-8383-b4c7726ff2d1" containerID="350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91" exitCode=0 Aug 04 16:53:06.791284 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.790836 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" event={"ID":"c7232160-2148-4f0d-8383-b4c7726ff2d1","Type":"ContainerDied","Data":"350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91"} Aug 04 16:53:06.791284 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.790850 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" Aug 04 16:53:06.791284 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.790863 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2" event={"ID":"c7232160-2148-4f0d-8383-b4c7726ff2d1","Type":"ContainerDied","Data":"429e3e9f8424e118ed30552849564d9aa01a749c3ec7aeedc16c046dd86b06eb"} Aug 04 16:53:06.791284 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.790891 2571 scope.go:117] "RemoveContainer" containerID="350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91" Aug 04 16:53:06.800127 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.800113 2571 scope.go:117] "RemoveContainer" containerID="350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91" Aug 04 16:53:06.800414 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:53:06.800395 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91\": container with ID starting with 350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91 not found: ID does not exist" containerID="350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91" Aug 04 16:53:06.800474 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.800422 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91"} err="failed to get container status \"350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91\": rpc error: code = NotFound desc = could not find container \"350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91\": container with ID starting with 350aea6dc69666325a9d7325733d3d06ea38d4068629054bf931c134c2c44a91 not found: ID does not exist" Aug 04 16:53:06.813167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.813147 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2"] Aug 04 16:53:06.815407 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:06.815384 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/sub-test-model-5a3cb8-kserve-5cb6ccb99f-prtq2"] Aug 04 16:53:07.363719 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:07.363683 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7232160-2148-4f0d-8383-b4c7726ff2d1" path="/var/lib/kubelet/pods/c7232160-2148-4f0d-8383-b4c7726ff2d1/volumes" Aug 04 16:53:07.982096 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:07.982062 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:53:07.982529 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:07.982109 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:53:07.994120 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:07.994095 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:53:08.814556 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:08.814528 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:53:19.085387 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.085348 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2"] Aug 04 16:53:19.085866 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.085743 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7232160-2148-4f0d-8383-b4c7726ff2d1" containerName="main" Aug 04 16:53:19.085866 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.085756 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7232160-2148-4f0d-8383-b4c7726ff2d1" containerName="main" Aug 04 16:53:19.085866 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.085837 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7232160-2148-4f0d-8383-b4c7726ff2d1" containerName="main" Aug 04 16:53:19.088943 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.088919 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.091891 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.091870 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"kube-root-ca.crt\"" Aug 04 16:53:19.093024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.093003 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"openshift-service-ca.crt\"" Aug 04 16:53:19.093125 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.093042 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"default-dockercfg-xcqvj\"" Aug 04 16:53:19.093350 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.093334 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a3cb8\"/\"rate-test-model-5a3cb8-kserve-self-signed-certs\"" Aug 04 16:53:19.098063 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.098041 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2"] Aug 04 16:53:19.113407 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.113359 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:53:19.159604 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.159579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tls-certs\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.159731 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.159617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qflwg\" (UniqueName: \"kubernetes.io/projected/5cd8dd06-91a3-49ce-9022-2366a7536cd9-kube-api-access-qflwg\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.159731 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.159651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-dshm\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.159731 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.159716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-model-cache\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.159843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.159788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tmp-dir\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.159888 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.159840 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-home\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261023 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.260991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-dshm\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261188 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261030 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-model-cache\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261188 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261079 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tmp-dir\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261188 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-home\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261188 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261137 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tls-certs\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261188 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261161 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qflwg\" (UniqueName: \"kubernetes.io/projected/5cd8dd06-91a3-49ce-9022-2366a7536cd9-kube-api-access-qflwg\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261541 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261518 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tmp-dir\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261615 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261593 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-home\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.261672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.261648 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-model-cache\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.263436 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.263414 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-dshm\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.263619 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.263604 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tls-certs\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.268969 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.268947 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qflwg\" (UniqueName: \"kubernetes.io/projected/5cd8dd06-91a3-49ce-9022-2366a7536cd9-kube-api-access-qflwg\") pod \"rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.399944 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.399859 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:19.532960 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.532932 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2"] Aug 04 16:53:19.534917 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:53:19.534889 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5cd8dd06_91a3_49ce_9022_2366a7536cd9.slice/crio-bd22828fdf7fde1c69886b49f3ccd533a1340d2e78200c3360c05a9e7c39a885 WatchSource:0}: Error finding container bd22828fdf7fde1c69886b49f3ccd533a1340d2e78200c3360c05a9e7c39a885: Status 404 returned error can't find the container with id bd22828fdf7fde1c69886b49f3ccd533a1340d2e78200c3360c05a9e7c39a885 Aug 04 16:53:19.864148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.864114 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" event={"ID":"5cd8dd06-91a3-49ce-9022-2366a7536cd9","Type":"ContainerStarted","Data":"7e78ad06bcd39983abde08b94e960555849e7b6e2e8c37f777ce659058b9b402"} Aug 04 16:53:19.864148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.864157 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" event={"ID":"5cd8dd06-91a3-49ce-9022-2366a7536cd9","Type":"ContainerStarted","Data":"bd22828fdf7fde1c69886b49f3ccd533a1340d2e78200c3360c05a9e7c39a885"} Aug 04 16:53:19.882445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:19.882401 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" podStartSLOduration=0.882388326 podStartE2EDuration="882.388326ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:53:19.881403583 +0000 UTC m=+3408.975003846" watchObservedRunningTime="2026-08-04 16:53:19.882388326 +0000 UTC m=+3408.975988573" Aug 04 16:53:21.682112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:21.682073 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:53:29.400473 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:29.400391 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:29.400473 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:29.400439 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:29.413027 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:29.413007 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:53:29.928038 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:53:29.928008 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:54:02.687802 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:02.687768 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:54:14.297876 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:14.297840 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:54:26.109997 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:26.109955 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:54:42.880855 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:42.880817 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:54:48.487249 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:48.487183 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn"] Aug 04 16:54:48.507162 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:48.507134 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:54:49.280053 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:49.280013 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2"] Aug 04 16:54:49.310273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:49.310246 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:54:54.980020 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:54:54.979933 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:55:03.505927 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.505863 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" podUID="0850def6-f349-4e9b-b92e-b9ee4274dfb2" containerName="main" containerID="cri-o://1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f" gracePeriod=45 Aug 04 16:55:03.760018 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.759965 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:55:03.828385 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828350 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tls-certs\") pod \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " Aug 04 16:55:03.828554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828408 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-home\") pod \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " Aug 04 16:55:03.828554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828449 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tmp-dir\") pod \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " Aug 04 16:55:03.828554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828487 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdb9d\" (UniqueName: \"kubernetes.io/projected/0850def6-f349-4e9b-b92e-b9ee4274dfb2-kube-api-access-xdb9d\") pod \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " Aug 04 16:55:03.828554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828509 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-dshm\") pod \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " Aug 04 16:55:03.828554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828539 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-model-cache\") pod \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\" (UID: \"0850def6-f349-4e9b-b92e-b9ee4274dfb2\") " Aug 04 16:55:03.828840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828724 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-home" (OuterVolumeSpecName: "home") pod "0850def6-f349-4e9b-b92e-b9ee4274dfb2" (UID: "0850def6-f349-4e9b-b92e-b9ee4274dfb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:03.828840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828740 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0850def6-f349-4e9b-b92e-b9ee4274dfb2" (UID: "0850def6-f349-4e9b-b92e-b9ee4274dfb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:03.828951 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828862 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "0850def6-f349-4e9b-b92e-b9ee4274dfb2" (UID: "0850def6-f349-4e9b-b92e-b9ee4274dfb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:03.828951 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828922 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:03.828951 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.828936 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:03.830545 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.830521 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-dshm" (OuterVolumeSpecName: "dshm") pod "0850def6-f349-4e9b-b92e-b9ee4274dfb2" (UID: "0850def6-f349-4e9b-b92e-b9ee4274dfb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:03.830645 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.830588 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0850def6-f349-4e9b-b92e-b9ee4274dfb2" (UID: "0850def6-f349-4e9b-b92e-b9ee4274dfb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:55:03.830645 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.830629 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0850def6-f349-4e9b-b92e-b9ee4274dfb2-kube-api-access-xdb9d" (OuterVolumeSpecName: "kube-api-access-xdb9d") pod "0850def6-f349-4e9b-b92e-b9ee4274dfb2" (UID: "0850def6-f349-4e9b-b92e-b9ee4274dfb2"). InnerVolumeSpecName "kube-api-access-xdb9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:55:03.929464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.929437 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0850def6-f349-4e9b-b92e-b9ee4274dfb2-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:03.929464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.929462 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdb9d\" (UniqueName: \"kubernetes.io/projected/0850def6-f349-4e9b-b92e-b9ee4274dfb2-kube-api-access-xdb9d\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:03.929616 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.929471 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:03.929616 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:03.929480 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0850def6-f349-4e9b-b92e-b9ee4274dfb2-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:04.299942 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.299873 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" podUID="5cd8dd06-91a3-49ce-9022-2366a7536cd9" containerName="main" containerID="cri-o://7e78ad06bcd39983abde08b94e960555849e7b6e2e8c37f777ce659058b9b402" gracePeriod=45 Aug 04 16:55:04.447088 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.447060 2571 generic.go:356] "Generic (PLEG): container finished" podID="5cd8dd06-91a3-49ce-9022-2366a7536cd9" containerID="7e78ad06bcd39983abde08b94e960555849e7b6e2e8c37f777ce659058b9b402" exitCode=0 Aug 04 16:55:04.447269 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.447138 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" event={"ID":"5cd8dd06-91a3-49ce-9022-2366a7536cd9","Type":"ContainerDied","Data":"7e78ad06bcd39983abde08b94e960555849e7b6e2e8c37f777ce659058b9b402"} Aug 04 16:55:04.448520 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.448495 2571 generic.go:356] "Generic (PLEG): container finished" podID="0850def6-f349-4e9b-b92e-b9ee4274dfb2" containerID="1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f" exitCode=0 Aug 04 16:55:04.448665 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.448528 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" event={"ID":"0850def6-f349-4e9b-b92e-b9ee4274dfb2","Type":"ContainerDied","Data":"1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f"} Aug 04 16:55:04.448665 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.448553 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" Aug 04 16:55:04.448665 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.448578 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn" event={"ID":"0850def6-f349-4e9b-b92e-b9ee4274dfb2","Type":"ContainerDied","Data":"9e655ccbc3f88494e22c0d3a38717747db524f77b885a67efae7af402491e010"} Aug 04 16:55:04.448665 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.448604 2571 scope.go:117] "RemoveContainer" containerID="1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f" Aug 04 16:55:04.457724 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.457709 2571 scope.go:117] "RemoveContainer" containerID="1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f" Aug 04 16:55:04.457958 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:55:04.457939 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f\": container with ID starting with 1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f not found: ID does not exist" containerID="1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f" Aug 04 16:55:04.458014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.457966 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f"} err="failed to get container status \"1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f\": rpc error: code = NotFound desc = could not find container \"1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f\": container with ID starting with 1ea41c4a600b525b7f9f8e691cfb4183904a4e56a26cceff9eb89cb9188b812f not found: ID does not exist" Aug 04 16:55:04.473478 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.473455 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn"] Aug 04 16:55:04.477347 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.477323 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-90d707/rate-test-model-90d707-kserve-7c6587c45f-s5xvn"] Aug 04 16:55:04.559949 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.559928 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:55:04.636273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636249 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-model-cache\") pod \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " Aug 04 16:55:04.636445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636294 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-home\") pod \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " Aug 04 16:55:04.636445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636329 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tls-certs\") pod \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " Aug 04 16:55:04.636445 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636401 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-dshm\") pod \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " Aug 04 16:55:04.636621 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636445 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qflwg\" (UniqueName: \"kubernetes.io/projected/5cd8dd06-91a3-49ce-9022-2366a7536cd9-kube-api-access-qflwg\") pod \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " Aug 04 16:55:04.636621 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636477 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tmp-dir\") pod \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\" (UID: \"5cd8dd06-91a3-49ce-9022-2366a7536cd9\") " Aug 04 16:55:04.636621 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636561 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5cd8dd06-91a3-49ce-9022-2366a7536cd9" (UID: "5cd8dd06-91a3-49ce-9022-2366a7536cd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:04.636621 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636575 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-home" (OuterVolumeSpecName: "home") pod "5cd8dd06-91a3-49ce-9022-2366a7536cd9" (UID: "5cd8dd06-91a3-49ce-9022-2366a7536cd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:04.636790 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636771 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5cd8dd06-91a3-49ce-9022-2366a7536cd9" (UID: "5cd8dd06-91a3-49ce-9022-2366a7536cd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:04.636854 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636816 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:04.636854 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.636829 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:04.638458 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.638426 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-dshm" (OuterVolumeSpecName: "dshm") pod "5cd8dd06-91a3-49ce-9022-2366a7536cd9" (UID: "5cd8dd06-91a3-49ce-9022-2366a7536cd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:55:04.638458 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.638426 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd8dd06-91a3-49ce-9022-2366a7536cd9-kube-api-access-qflwg" (OuterVolumeSpecName: "kube-api-access-qflwg") pod "5cd8dd06-91a3-49ce-9022-2366a7536cd9" (UID: "5cd8dd06-91a3-49ce-9022-2366a7536cd9"). InnerVolumeSpecName "kube-api-access-qflwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:55:04.638634 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.638472 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5cd8dd06-91a3-49ce-9022-2366a7536cd9" (UID: "5cd8dd06-91a3-49ce-9022-2366a7536cd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:55:04.737910 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.737886 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:04.737910 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.737906 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:04.738051 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.737914 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qflwg\" (UniqueName: \"kubernetes.io/projected/5cd8dd06-91a3-49ce-9022-2366a7536cd9-kube-api-access-qflwg\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:04.738051 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:04.737924 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd8dd06-91a3-49ce-9022-2366a7536cd9-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:05.363453 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.363406 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0850def6-f349-4e9b-b92e-b9ee4274dfb2" path="/var/lib/kubelet/pods/0850def6-f349-4e9b-b92e-b9ee4274dfb2/volumes" Aug 04 16:55:05.455750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.455725 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" Aug 04 16:55:05.455909 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.455728 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2" event={"ID":"5cd8dd06-91a3-49ce-9022-2366a7536cd9","Type":"ContainerDied","Data":"bd22828fdf7fde1c69886b49f3ccd533a1340d2e78200c3360c05a9e7c39a885"} Aug 04 16:55:05.455909 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.455856 2571 scope.go:117] "RemoveContainer" containerID="7e78ad06bcd39983abde08b94e960555849e7b6e2e8c37f777ce659058b9b402" Aug 04 16:55:05.473743 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.473721 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2"] Aug 04 16:55:05.477387 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.477366 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5a3cb8/rate-test-model-5a3cb8-kserve-98df5fbcf-vr9c2"] Aug 04 16:55:05.897697 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.897664 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:05.898081 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.898068 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0850def6-f349-4e9b-b92e-b9ee4274dfb2" containerName="main" Aug 04 16:55:05.898081 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.898083 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0850def6-f349-4e9b-b92e-b9ee4274dfb2" containerName="main" Aug 04 16:55:05.898156 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.898090 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd8dd06-91a3-49ce-9022-2366a7536cd9" containerName="main" Aug 04 16:55:05.898156 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.898095 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd8dd06-91a3-49ce-9022-2366a7536cd9" containerName="main" Aug 04 16:55:05.898243 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.898161 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0850def6-f349-4e9b-b92e-b9ee4274dfb2" containerName="main" Aug 04 16:55:05.898243 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.898172 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd8dd06-91a3-49ce-9022-2366a7536cd9" containerName="main" Aug 04 16:55:05.902650 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.902633 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:05.905737 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.905714 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gstnk\"" Aug 04 16:55:05.908592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.908565 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:05.947256 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:05.947233 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfs7g\" (UniqueName: \"kubernetes.io/projected/61c4df42-816f-465e-bd79-ae45551c5379-kube-api-access-bfs7g\") pod \"test-curl-7355-140050664075600\" (UID: \"61c4df42-816f-465e-bd79-ae45551c5379\") " pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:06.047839 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:06.047808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfs7g\" (UniqueName: \"kubernetes.io/projected/61c4df42-816f-465e-bd79-ae45551c5379-kube-api-access-bfs7g\") pod \"test-curl-7355-140050664075600\" (UID: \"61c4df42-816f-465e-bd79-ae45551c5379\") " pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:06.056351 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:06.056320 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfs7g\" (UniqueName: \"kubernetes.io/projected/61c4df42-816f-465e-bd79-ae45551c5379-kube-api-access-bfs7g\") pod \"test-curl-7355-140050664075600\" (UID: \"61c4df42-816f-465e-bd79-ae45551c5379\") " pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:06.214089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:06.213998 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:06.345468 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:06.345440 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:06.347503 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:55:06.347468 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61c4df42_816f_465e_bd79_ae45551c5379.slice/crio-5c49dc39ca91d942730d365bf896d935af8fbfba0d2f38c93b56e8eed5be30a0 WatchSource:0}: Error finding container 5c49dc39ca91d942730d365bf896d935af8fbfba0d2f38c93b56e8eed5be30a0: Status 404 returned error can't find the container with id 5c49dc39ca91d942730d365bf896d935af8fbfba0d2f38c93b56e8eed5be30a0 Aug 04 16:55:06.348843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:06.348824 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:55:06.467768 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:06.467688 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664075600" event={"ID":"61c4df42-816f-465e-bd79-ae45551c5379","Type":"ContainerStarted","Data":"5c49dc39ca91d942730d365bf896d935af8fbfba0d2f38c93b56e8eed5be30a0"} Aug 04 16:55:07.363532 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:07.363500 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd8dd06-91a3-49ce-9022-2366a7536cd9" path="/var/lib/kubelet/pods/5cd8dd06-91a3-49ce-9022-2366a7536cd9/volumes" Aug 04 16:55:07.474894 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:07.474861 2571 generic.go:356] "Generic (PLEG): container finished" podID="61c4df42-816f-465e-bd79-ae45551c5379" containerID="cf6b1b7c7acf48669fd386b318f80fa5cb7bdd1d35364480d23d9b3f5d09bf41" exitCode=0 Aug 04 16:55:07.475052 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:07.474911 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7355-140050664075600" event={"ID":"61c4df42-816f-465e-bd79-ae45551c5379","Type":"ContainerDied","Data":"cf6b1b7c7acf48669fd386b318f80fa5cb7bdd1d35364480d23d9b3f5d09bf41"} Aug 04 16:55:08.615939 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:08.615912 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:08.645087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:08.645062 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7355-140050664075600_61c4df42-816f-465e-bd79-ae45551c5379/test-curl-7355-140050664075600/0.log" Aug 04 16:55:08.671465 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:08.671439 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfs7g\" (UniqueName: \"kubernetes.io/projected/61c4df42-816f-465e-bd79-ae45551c5379-kube-api-access-bfs7g\") pod \"61c4df42-816f-465e-bd79-ae45551c5379\" (UID: \"61c4df42-816f-465e-bd79-ae45551c5379\") " Aug 04 16:55:08.673747 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:08.673721 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c4df42-816f-465e-bd79-ae45551c5379-kube-api-access-bfs7g" (OuterVolumeSpecName: "kube-api-access-bfs7g") pod "61c4df42-816f-465e-bd79-ae45551c5379" (UID: "61c4df42-816f-465e-bd79-ae45551c5379"). InnerVolumeSpecName "kube-api-access-bfs7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:55:08.674484 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:08.674460 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:08.680062 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:08.680043 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7355-140050664075600"] Aug 04 16:55:08.773007 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:08.772950 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfs7g\" (UniqueName: \"kubernetes.io/projected/61c4df42-816f-465e-bd79-ae45551c5379-kube-api-access-bfs7g\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:55:09.246145 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.246092 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:09.363096 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.363063 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c4df42-816f-465e-bd79-ae45551c5379" path="/var/lib/kubelet/pods/61c4df42-816f-465e-bd79-ae45551c5379/volumes" Aug 04 16:55:09.491867 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.491834 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7355-140050664075600" Aug 04 16:55:09.492039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.491901 2571 scope.go:117] "RemoveContainer" containerID="cf6b1b7c7acf48669fd386b318f80fa5cb7bdd1d35364480d23d9b3f5d09bf41" Aug 04 16:55:09.564419 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.564387 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9"] Aug 04 16:55:09.564973 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.564952 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c4df42-816f-465e-bd79-ae45551c5379" containerName="test-curl-7355-140050664075600" Aug 04 16:55:09.565024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.564978 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c4df42-816f-465e-bd79-ae45551c5379" containerName="test-curl-7355-140050664075600" Aug 04 16:55:09.565109 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.565097 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c4df42-816f-465e-bd79-ae45551c5379" containerName="test-curl-7355-140050664075600" Aug 04 16:55:09.569987 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.569969 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.574138 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.574118 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bbe0dc-openshift-default-dockercfg-9j6xw\"" Aug 04 16:55:09.583846 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.583806 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9"] Aug 04 16:55:09.680562 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680524 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-envoy\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-certs\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-credential-socket\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680733 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istiod-ca-cert\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680757 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-socket\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680789 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-podinfo\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680871 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-data\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680914 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9wtg\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-kube-api-access-p9wtg\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.680968 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.680945 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-token\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.706129 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.706102 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:09.781895 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.781816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9wtg\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-kube-api-access-p9wtg\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.781895 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.781870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-token\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.781908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-envoy\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.781971 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-certs\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-credential-socket\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782043 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istiod-ca-cert\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-socket\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782116 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782102 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-podinfo\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782429 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782164 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-data\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782501 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782481 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-certs\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782557 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782527 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-credential-socket\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782625 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782585 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-socket\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782683 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782670 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-data\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.782883 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.782861 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istiod-ca-cert\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.784423 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.784398 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-envoy\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.784640 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.784620 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-podinfo\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.791307 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.791285 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-token\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.791507 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.791489 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9wtg\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-kube-api-access-p9wtg\") pod \"e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:09.809161 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.809138 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:09.879644 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:09.879619 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:10.045165 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:10.045090 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9"] Aug 04 16:55:10.048309 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:55:10.048273 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70e46f4_30aa_4baf_bf15_c3513c8e602e.slice/crio-5e1a78d98892ee310e0f3eeb3e8950be66736090522bf1e2c619330340d128ed WatchSource:0}: Error finding container 5e1a78d98892ee310e0f3eeb3e8950be66736090522bf1e2c619330340d128ed: Status 404 returned error can't find the container with id 5e1a78d98892ee310e0f3eeb3e8950be66736090522bf1e2c619330340d128ed Aug 04 16:55:10.498631 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:10.498596 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" event={"ID":"f70e46f4-30aa-4baf-bf15-c3513c8e602e","Type":"ContainerStarted","Data":"5e1a78d98892ee310e0f3eeb3e8950be66736090522bf1e2c619330340d128ed"} Aug 04 16:55:12.753927 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:12.753891 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:12.754179 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:12.753964 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:12.754179 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:12.753994 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:13.520248 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:13.520198 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" event={"ID":"f70e46f4-30aa-4baf-bf15-c3513c8e602e","Type":"ContainerStarted","Data":"4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22"} Aug 04 16:55:13.543115 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:13.543064 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podStartSLOduration=1.8399895069999999 podStartE2EDuration="4.543049642s" totalImagesPullingTime="2.703060135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:55:13.539993554 +0000 UTC m=+3522.633593814" watchObservedRunningTime="2026-08-04 16:55:13.543049642 +0000 UTC m=+3522.636649889" Aug 04 16:55:13.880519 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:13.880485 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:13.885310 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:13.885288 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:14.526702 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:14.526671 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:14.527821 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:14.527798 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:15.504166 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:15.504129 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:15.607684 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:15.607627 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:16.707637 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:16.707588 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:16.806332 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:16.806283 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:17.214403 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:17.214357 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:17.315614 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:17.315579 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:19.189015 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:19.188980 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:55:29.528574 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.528478 2571 patch_prober.go:28] interesting pod/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Aug 04 16:55:29.528574 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.528540 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Aug 04 16:55:29.614639 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.614607 2571 generic.go:356] "Generic (PLEG): container finished" podID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerID="4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22" exitCode=0 Aug 04 16:55:29.614822 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.614651 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" event={"ID":"f70e46f4-30aa-4baf-bf15-c3513c8e602e","Type":"ContainerDied","Data":"4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22"} Aug 04 16:55:29.615041 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.615026 2571 scope.go:117] "RemoveContainer" containerID="4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22" Aug 04 16:55:29.617071 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.617038 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:29.617190 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.617127 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:29.617190 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:29.617176 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:30.623239 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:30.622388 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" event={"ID":"f70e46f4-30aa-4baf-bf15-c3513c8e602e","Type":"ContainerStarted","Data":"448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65"} Aug 04 16:55:30.880468 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:30.880376 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:30.880608 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:30.880587 2571 patch_prober.go:28] interesting pod/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Aug 04 16:55:30.880662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:30.880634 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Aug 04 16:55:31.880478 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:31.880434 2571 patch_prober.go:28] interesting pod/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Aug 04 16:55:31.880904 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:31.880511 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Aug 04 16:55:32.880450 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:32.880412 2571 patch_prober.go:28] interesting pod/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Aug 04 16:55:32.880632 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:32.880467 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Aug 04 16:55:33.880278 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:33.880238 2571 patch_prober.go:28] interesting pod/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Aug 04 16:55:33.880473 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:33.880298 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Aug 04 16:55:34.880174 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:34.880138 2571 patch_prober.go:28] interesting pod/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Aug 04 16:55:34.880603 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:34.880192 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Aug 04 16:55:35.884565 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:35.884532 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:35.884998 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:35.884790 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:35.885482 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:35.885461 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:55:40.550632 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:40.550596 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:41.255842 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.255809 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:41.558941 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.558912 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j"] Aug 04 16:55:41.566744 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.566717 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.569748 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.569723 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-54f057-openshift-default-dockercfg-745kh\"" Aug 04 16:55:41.573214 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.573174 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j"] Aug 04 16:55:41.579663 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.579627 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:41.685679 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685641 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-socket\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.685864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685702 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-token\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.685864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685722 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-certs\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.685864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685759 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-credential-socket\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.685864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685778 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfsgt\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-kube-api-access-hfsgt\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.685864 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9082291c-c94c-4214-bbd8-c2edab5b56c6-istiod-ca-cert\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.686039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685872 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-podinfo\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.686039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685893 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-envoy\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.686039 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.685953 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-data\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.786812 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.786762 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-certs\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.786978 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.786830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-credential-socket\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.786978 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.786861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfsgt\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-kube-api-access-hfsgt\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.786978 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.786882 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9082291c-c94c-4214-bbd8-c2edab5b56c6-istiod-ca-cert\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.786978 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.786902 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-podinfo\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.786978 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.786926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-envoy\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787295 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.786987 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-data\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787295 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.787031 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-socket\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787295 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.787097 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-token\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787465 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.787305 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-credential-socket\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787645 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.787614 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-certs\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787778 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.787679 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-data\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787778 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.787685 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-socket\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.787860 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.787837 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9082291c-c94c-4214-bbd8-c2edab5b56c6-istiod-ca-cert\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.789364 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.789344 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-envoy\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.789583 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.789565 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-podinfo\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.795611 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.795576 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfsgt\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-kube-api-access-hfsgt\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.795740 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.795711 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-token\") pod \"e2e-ipp-b-54f057-openshift-default-898995c54-cn76j\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:41.879904 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:41.879809 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:42.032905 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.032873 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j"] Aug 04 16:55:42.034261 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:55:42.034230 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9082291c_c94c_4214_bbd8_c2edab5b56c6.slice/crio-9ce0a3fe06bd1cb8264f0523b82460bf87d7386ec02c8465ca66b68f58281a61 WatchSource:0}: Error finding container 9ce0a3fe06bd1cb8264f0523b82460bf87d7386ec02c8465ca66b68f58281a61: Status 404 returned error can't find the container with id 9ce0a3fe06bd1cb8264f0523b82460bf87d7386ec02c8465ca66b68f58281a61 Aug 04 16:55:42.037051 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.037005 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:42.037154 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.037093 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:42.037154 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.037136 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:42.523597 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.523559 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:42.701578 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.701546 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" event={"ID":"9082291c-c94c-4214-bbd8-c2edab5b56c6","Type":"ContainerStarted","Data":"bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326"} Aug 04 16:55:42.701963 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.701597 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" event={"ID":"9082291c-c94c-4214-bbd8-c2edab5b56c6","Type":"ContainerStarted","Data":"9ce0a3fe06bd1cb8264f0523b82460bf87d7386ec02c8465ca66b68f58281a61"} Aug 04 16:55:42.724517 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.724471 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podStartSLOduration=1.724456284 podStartE2EDuration="1.724456284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:55:42.722517566 +0000 UTC m=+3551.816117840" watchObservedRunningTime="2026-08-04 16:55:42.724456284 +0000 UTC m=+3551.818056531" Aug 04 16:55:42.880501 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:42.880467 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:43.885179 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:43.885148 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:44.685734 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:44.685698 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:44.712799 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:44.712768 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:44.713757 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:44.713735 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:55:44.787735 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:44.787703 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:45.994707 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:45.994673 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:46.520410 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:46.520372 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:46.631560 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:46.631528 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:46.741037 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:46.741000 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:55:49.180835 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:49.180803 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:55:59.714259 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.714190 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Aug 04 16:55:59.714675 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.714292 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Aug 04 16:55:59.806581 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.806549 2571 generic.go:356] "Generic (PLEG): container finished" podID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerID="bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326" exitCode=0 Aug 04 16:55:59.806720 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.806597 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" event={"ID":"9082291c-c94c-4214-bbd8-c2edab5b56c6","Type":"ContainerDied","Data":"bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326"} Aug 04 16:55:59.807037 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.807021 2571 scope.go:117] "RemoveContainer" containerID="bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326" Aug 04 16:55:59.808087 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.808055 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:59.808167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.808126 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:55:59.808167 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:55:59.808159 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:56:00.813669 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:00.813632 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" event={"ID":"9082291c-c94c-4214-bbd8-c2edab5b56c6","Type":"ContainerStarted","Data":"d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88"} Aug 04 16:56:00.881054 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:00.881015 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:56:00.881323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:00.881298 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Aug 04 16:56:00.881389 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:00.881358 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Aug 04 16:56:01.880441 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:01.880406 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Aug 04 16:56:01.880840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:01.880463 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Aug 04 16:56:02.880768 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:02.880733 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Aug 04 16:56:02.881165 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:02.880795 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Aug 04 16:56:03.880649 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:03.880612 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Aug 04 16:56:03.880835 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:03.880665 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Aug 04 16:56:04.880975 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:04.880935 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Aug 04 16:56:04.881413 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:04.880996 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Aug 04 16:56:05.880725 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:05.880688 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Aug 04 16:56:05.880908 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:05.880761 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Aug 04 16:56:06.884435 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:06.884403 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:56:06.884929 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:06.884655 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:56:06.885443 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:06.885425 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:56:09.726652 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:09.726617 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:56:41.685450 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.685411 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d"] Aug 04 16:56:41.687878 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.687862 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.690451 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.690431 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bbe0dc\"/\"kube-root-ca.crt\"" Aug 04 16:56:41.691554 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.691528 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bbe0dc\"/\"ipp-route-bbe0dc-kserve-self-signed-certs\"" Aug 04 16:56:41.691672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.691612 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bbe0dc\"/\"openshift-service-ca.crt\"" Aug 04 16:56:41.691672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.691633 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bbe0dc\"/\"default-dockercfg-tvmtb\"" Aug 04 16:56:41.699345 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.699316 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d"] Aug 04 16:56:41.770118 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.770087 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-dshm\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.770314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.770125 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tls-certs\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.770314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.770189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-model-cache\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.770314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.770252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkjbn\" (UniqueName: \"kubernetes.io/projected/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-kube-api-access-qkjbn\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.770314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.770272 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-home\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.770314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.770293 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tmp-dir\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.782414 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.782378 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:56:41.871661 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.871629 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-dshm\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.871843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.871667 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tls-certs\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.871843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.871717 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-model-cache\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.871843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.871751 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkjbn\" (UniqueName: \"kubernetes.io/projected/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-kube-api-access-qkjbn\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.871843 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.871819 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-home\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.872020 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.871855 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tmp-dir\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.872184 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.872163 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-model-cache\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.872298 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.872230 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-home\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.872381 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.872342 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tmp-dir\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.874330 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.874310 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-dshm\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.874427 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.874381 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tls-certs\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.880750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.880722 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkjbn\" (UniqueName: \"kubernetes.io/projected/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-kube-api-access-qkjbn\") pod \"ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:41.997037 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:41.996932 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:42.142782 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:42.142751 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d"] Aug 04 16:56:42.144009 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:56:42.143980 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56e51a76_db55_4f0c_b0cb_4e4c526a5bb5.slice/crio-9ad182d8a69d9ed1ad5ad9b16823464a92e89fec694784036af0d424d5c167f6 WatchSource:0}: Error finding container 9ad182d8a69d9ed1ad5ad9b16823464a92e89fec694784036af0d424d5c167f6: Status 404 returned error can't find the container with id 9ad182d8a69d9ed1ad5ad9b16823464a92e89fec694784036af0d424d5c167f6 Aug 04 16:56:43.065548 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:43.065511 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" event={"ID":"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5","Type":"ContainerStarted","Data":"2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15"} Aug 04 16:56:43.065548 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:43.065553 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" event={"ID":"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5","Type":"ContainerStarted","Data":"9ad182d8a69d9ed1ad5ad9b16823464a92e89fec694784036af0d424d5c167f6"} Aug 04 16:56:44.677194 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:44.677130 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" podStartSLOduration=3.677111581 podStartE2EDuration="3.677111581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:56:43.088469513 +0000 UTC m=+3612.182069766" watchObservedRunningTime="2026-08-04 16:56:44.677111581 +0000 UTC m=+3613.770711828" Aug 04 16:56:44.679137 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:44.679110 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:56:51.997362 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:51.997327 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:51.997362 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:51.997375 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:52.009885 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:52.009859 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:56:52.133611 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:56:52.133581 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:57:05.981535 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:05.981498 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:57:17.737270 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:17.737234 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:18.045812 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.045727 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w"] Aug 04 16:57:18.049359 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.049337 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.051982 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.051960 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-4dd86f-openshift-default-dockercfg-54srw\"" Aug 04 16:57:18.060933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.060910 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w"] Aug 04 16:57:18.074340 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.074305 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:18.129516 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-token\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129666 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-envoy\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129666 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-certs\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129666 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129626 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-data\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7l47\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-kube-api-access-d7l47\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129717 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-credential-socket\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129738 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-podinfo\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129758 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e9deff-9bc1-47d7-9841-1d5a196da819-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.129820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.129775 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-socket\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231276 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7l47\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-kube-api-access-d7l47\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-credential-socket\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231314 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-podinfo\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231334 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e9deff-9bc1-47d7-9841-1d5a196da819-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231352 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-socket\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231464 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231445 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-token\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231729 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231480 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-envoy\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231729 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-certs\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231729 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231552 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-data\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231902 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231828 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-credential-socket\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.231962 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231949 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-data\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.232017 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.231960 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-socket\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.232108 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.232081 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-certs\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.232151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.232115 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e9deff-9bc1-47d7-9841-1d5a196da819-istiod-ca-cert\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.233770 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.233749 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-envoy\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.233937 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.233919 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-podinfo\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.239739 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.239709 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7l47\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-kube-api-access-d7l47\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.239916 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.239898 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-token\") pod \"e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.264533 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.264508 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:18.361287 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.361242 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:18.499884 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.499811 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w"] Aug 04 16:57:18.500985 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:57:18.500956 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e9deff_9bc1_47d7_9841_1d5a196da819.slice/crio-2fec0f7621ee01515cf7a05102388ea1d648a4007605b4bbb49b4fd0f72beaf2 WatchSource:0}: Error finding container 2fec0f7621ee01515cf7a05102388ea1d648a4007605b4bbb49b4fd0f72beaf2: Status 404 returned error can't find the container with id 2fec0f7621ee01515cf7a05102388ea1d648a4007605b4bbb49b4fd0f72beaf2 Aug 04 16:57:18.503217 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.503177 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:57:18.503297 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.503257 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:57:18.503297 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:18.503286 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:57:19.292848 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:19.292804 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" event={"ID":"d2e9deff-9bc1-47d7-9841-1d5a196da819","Type":"ContainerStarted","Data":"59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9"} Aug 04 16:57:19.293269 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:19.292856 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" event={"ID":"d2e9deff-9bc1-47d7-9841-1d5a196da819","Type":"ContainerStarted","Data":"2fec0f7621ee01515cf7a05102388ea1d648a4007605b4bbb49b4fd0f72beaf2"} Aug 04 16:57:19.314426 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:19.314378 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podStartSLOduration=1.314362603 podStartE2EDuration="1.314362603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:57:19.312465509 +0000 UTC m=+3648.406065780" watchObservedRunningTime="2026-08-04 16:57:19.314362603 +0000 UTC m=+3648.407962850" Aug 04 16:57:19.363027 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:19.362998 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:19.366498 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:19.366473 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:20.298548 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:20.298515 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:20.299356 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:20.299338 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:21.313172 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:21.313120 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:21.408573 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:21.408537 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:22.565842 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:22.565810 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:22.813816 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:22.813782 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:23.126267 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:23.126233 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:23.232038 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:23.232004 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:27.782220 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:27.782175 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:57:36.340379 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:36.340339 2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Aug 04 16:57:36.340760 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:36.340413 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Aug 04 16:57:37.399710 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:37.399681 2571 generic.go:356] "Generic (PLEG): container finished" podID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerID="59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9" exitCode=0 Aug 04 16:57:37.400053 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:37.399754 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" event={"ID":"d2e9deff-9bc1-47d7-9841-1d5a196da819","Type":"ContainerDied","Data":"59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9"} Aug 04 16:57:37.400179 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:37.400165 2571 scope.go:117] "RemoveContainer" containerID="59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9" Aug 04 16:57:37.401378 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:37.401350 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:57:37.401433 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:37.401418 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:57:37.401479 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:37.401451 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Aug 04 16:57:38.407622 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:38.407589 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" event={"ID":"d2e9deff-9bc1-47d7-9841-1d5a196da819","Type":"ContainerStarted","Data":"3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7"} Aug 04 16:57:39.361693 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:39.361657 2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Aug 04 16:57:39.361881 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:39.361723 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Aug 04 16:57:39.364009 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:39.363978 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:39.756865 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:39.756772 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:39.811329 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:39.811299 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:40.361960 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:40.361921 2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Aug 04 16:57:40.362137 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:40.361988 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Aug 04 16:57:41.361679 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:41.361649 2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Aug 04 16:57:41.362111 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:41.361699 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Aug 04 16:57:42.362557 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:42.362522 2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Aug 04 16:57:42.362971 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:42.362591 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Aug 04 16:57:43.361941 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:43.361906 2571 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Aug 04 16:57:43.362151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:43.361987 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Aug 04 16:57:44.365340 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:44.365301 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:44.365835 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:44.365648 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:44.366173 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:44.366154 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:57:46.346077 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:46.346042 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:47.063911 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.063878 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5"] Aug 04 16:57:47.066717 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.066702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" Aug 04 16:57:47.069612 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.069590 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:57:47.069719 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.069591 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:57:47.075854 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.075831 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5"] Aug 04 16:57:47.127617 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.127580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146de1c6-d4aa-4f2c-967f-1b8775635979-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5\" (UID: \"146de1c6-d4aa-4f2c-967f-1b8775635979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" Aug 04 16:57:47.229140 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.229099 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146de1c6-d4aa-4f2c-967f-1b8775635979-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5\" (UID: \"146de1c6-d4aa-4f2c-967f-1b8775635979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" Aug 04 16:57:47.229773 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.229749 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146de1c6-d4aa-4f2c-967f-1b8775635979-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5\" (UID: \"146de1c6-d4aa-4f2c-967f-1b8775635979\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" Aug 04 16:57:47.379174 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.379093 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" Aug 04 16:57:47.512934 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:47.512903 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5"] Aug 04 16:57:47.514307 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:57:47.514281 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146de1c6_d4aa_4f2c_967f_1b8775635979.slice/crio-af95b0f2ae88f8198672093717e6c817200e5c37b5002871ed1e6fd88d850deb WatchSource:0}: Error finding container af95b0f2ae88f8198672093717e6c817200e5c37b5002871ed1e6fd88d850deb: Status 404 returned error can't find the container with id af95b0f2ae88f8198672093717e6c817200e5c37b5002871ed1e6fd88d850deb Aug 04 16:57:48.488943 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:48.488862 2571 generic.go:356] "Generic (PLEG): container finished" podID="146de1c6-d4aa-4f2c-967f-1b8775635979" containerID="7baf7229490f20551862f019c5e0da9765697cc23bc92724cbe2aa674f541071" exitCode=0 Aug 04 16:57:48.489314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:48.488944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" event={"ID":"146de1c6-d4aa-4f2c-967f-1b8775635979","Type":"ContainerDied","Data":"7baf7229490f20551862f019c5e0da9765697cc23bc92724cbe2aa674f541071"} Aug 04 16:57:48.489314 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:48.489003 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" event={"ID":"146de1c6-d4aa-4f2c-967f-1b8775635979","Type":"ContainerStarted","Data":"af95b0f2ae88f8198672093717e6c817200e5c37b5002871ed1e6fd88d850deb"} Aug 04 16:57:49.629681 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:49.629654 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" Aug 04 16:57:49.751870 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:49.751792 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146de1c6-d4aa-4f2c-967f-1b8775635979-maas-api-service-ca\") pod \"146de1c6-d4aa-4f2c-967f-1b8775635979\" (UID: \"146de1c6-d4aa-4f2c-967f-1b8775635979\") " Aug 04 16:57:49.752102 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:49.752081 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146de1c6-d4aa-4f2c-967f-1b8775635979-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "146de1c6-d4aa-4f2c-967f-1b8775635979" (UID: "146de1c6-d4aa-4f2c-967f-1b8775635979"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:57:49.853088 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:49.853055 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/146de1c6-d4aa-4f2c-967f-1b8775635979-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:57:50.501568 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:50.501543 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" Aug 04 16:57:50.501734 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:50.501581 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5" event={"ID":"146de1c6-d4aa-4f2c-967f-1b8775635979","Type":"ContainerDied","Data":"af95b0f2ae88f8198672093717e6c817200e5c37b5002871ed1e6fd88d850deb"} Aug 04 16:57:50.501734 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:50.501630 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af95b0f2ae88f8198672093717e6c817200e5c37b5002871ed1e6fd88d850deb" Aug 04 16:57:53.269532 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:53.269461 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:53.343354 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:53.343323 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:53.410793 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:53.410757 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:53.450513 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:53.450485 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5"] Aug 04 16:57:53.452840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:53.452818 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-4dd86f-fgrw5"] Aug 04 16:57:53.511328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:53.511295 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:53.819138 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:53.819093 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:55.363513 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:55.363485 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146de1c6-d4aa-4f2c-967f-1b8775635979" path="/var/lib/kubelet/pods/146de1c6-d4aa-4f2c-967f-1b8775635979/volumes" Aug 04 16:57:56.381749 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:56.381713 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:57:59.689814 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:59.689782 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:59.753339 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:59.753310 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:57:59.941344 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:59.941265 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w"] Aug 04 16:57:59.941506 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:59.941481 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" containerID="cri-o://3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7" gracePeriod=30 Aug 04 16:57:59.966449 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:57:59.966425 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:00.202223 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:00.202092 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:05.191067 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.191041 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:58:05.298783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.298695 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-credential-socket\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.298783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.298742 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-socket\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.298783 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.298776 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-podinfo\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.299050 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.298821 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e9deff-9bc1-47d7-9841-1d5a196da819-istiod-ca-cert\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.299050 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.298864 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-token\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.299050 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.298940 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7l47\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-kube-api-access-d7l47\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.299050 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.298987 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:05.299050 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299017 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:05.299050 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299039 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-certs\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.299378 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299073 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-envoy\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.299378 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299121 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-data\") pod \"d2e9deff-9bc1-47d7-9841-1d5a196da819\" (UID: \"d2e9deff-9bc1-47d7-9841-1d5a196da819\") " Aug 04 16:58:05.299378 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299301 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:05.299378 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299338 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e9deff-9bc1-47d7-9841-1d5a196da819-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:58:05.299591 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299457 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:05.299591 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299571 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.299591 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299591 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-data\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.299750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299604 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-credential-socket\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.299750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299618 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-workload-socket\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.299750 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.299630 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2e9deff-9bc1-47d7-9841-1d5a196da819-istiod-ca-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.301442 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.301417 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:58:05.301442 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.301426 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:05.301592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.301475 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 16:58:05.301592 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.301477 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-kube-api-access-d7l47" (OuterVolumeSpecName: "kube-api-access-d7l47") pod "d2e9deff-9bc1-47d7-9841-1d5a196da819" (UID: "d2e9deff-9bc1-47d7-9841-1d5a196da819"). InnerVolumeSpecName "kube-api-access-d7l47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:58:05.400336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.400306 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7l47\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-kube-api-access-d7l47\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.400336 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.400335 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-envoy\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.400494 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.400351 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-podinfo\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.400494 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.400363 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2e9deff-9bc1-47d7-9841-1d5a196da819-istio-token\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:05.591319 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.591284 2571 generic.go:356] "Generic (PLEG): container finished" podID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerID="3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7" exitCode=0 Aug 04 16:58:05.591491 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.591348 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" Aug 04 16:58:05.591491 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.591368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" event={"ID":"d2e9deff-9bc1-47d7-9841-1d5a196da819","Type":"ContainerDied","Data":"3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7"} Aug 04 16:58:05.591491 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.591444 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w" event={"ID":"d2e9deff-9bc1-47d7-9841-1d5a196da819","Type":"ContainerDied","Data":"2fec0f7621ee01515cf7a05102388ea1d648a4007605b4bbb49b4fd0f72beaf2"} Aug 04 16:58:05.591491 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.591465 2571 scope.go:117] "RemoveContainer" containerID="3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7" Aug 04 16:58:05.600803 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.600789 2571 scope.go:117] "RemoveContainer" containerID="59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9" Aug 04 16:58:05.609754 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.609735 2571 scope.go:117] "RemoveContainer" containerID="3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7" Aug 04 16:58:05.610018 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:58:05.609999 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7\": container with ID starting with 3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7 not found: ID does not exist" containerID="3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7" Aug 04 16:58:05.610080 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.610036 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7"} err="failed to get container status \"3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7\": rpc error: code = NotFound desc = could not find container \"3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7\": container with ID starting with 3767401ef010923a8c0565c019ea42617ef4222d97f182020245df52c896eea7 not found: ID does not exist" Aug 04 16:58:05.610080 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.610056 2571 scope.go:117] "RemoveContainer" containerID="59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9" Aug 04 16:58:05.610325 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:58:05.610307 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9\": container with ID starting with 59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9 not found: ID does not exist" containerID="59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9" Aug 04 16:58:05.610390 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.610330 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9"} err="failed to get container status \"59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9\": rpc error: code = NotFound desc = could not find container \"59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9\": container with ID starting with 59a049661cd99aafa9675e0d6b7d3cdae92daec39a672b261b3fbf5de23f1ed9 not found: ID does not exist" Aug 04 16:58:05.611169 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.611149 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w"] Aug 04 16:58:05.615715 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:05.615692 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-4dd86f-openshift-default-5897ccc57b-9gl9w"] Aug 04 16:58:07.363900 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:07.363868 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" path="/var/lib/kubelet/pods/d2e9deff-9bc1-47d7-9841-1d5a196da819/volumes" Aug 04 16:58:17.579936 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:17.579902 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:58:19.369124 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:19.369095 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:19.436240 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:19.436187 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:19.503323 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:19.503285 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:19.564447 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:19.564404 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:19.943820 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:19.943782 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:24.927254 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:24.927218 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d"] Aug 04 16:58:24.966353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:24.966325 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:58:25.479145 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:25.479111 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:58:39.946408 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:39.946348 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" podUID="56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" containerName="main" containerID="cri-o://2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15" gracePeriod=45 Aug 04 16:58:40.191186 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.191160 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:58:40.315827 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.315792 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-home\") pod \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " Aug 04 16:58:40.316021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.315850 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-dshm\") pod \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " Aug 04 16:58:40.316021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.315901 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tls-certs\") pod \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " Aug 04 16:58:40.316021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.315951 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-model-cache\") pod \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " Aug 04 16:58:40.316197 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.316051 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkjbn\" (UniqueName: \"kubernetes.io/projected/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-kube-api-access-qkjbn\") pod \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " Aug 04 16:58:40.316197 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.316094 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tmp-dir\") pod \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\" (UID: \"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5\") " Aug 04 16:58:40.316197 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.316125 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-home" (OuterVolumeSpecName: "home") pod "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" (UID: "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:40.316430 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.316384 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" (UID: "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:40.316531 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.316495 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-home\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:40.316531 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.316520 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-model-cache\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:40.316732 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.316660 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" (UID: "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:40.318977 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.318949 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-dshm" (OuterVolumeSpecName: "dshm") pod "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" (UID: "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:58:40.319117 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.318992 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" (UID: "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:58:40.319117 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.319010 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-kube-api-access-qkjbn" (OuterVolumeSpecName: "kube-api-access-qkjbn") pod "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" (UID: "56e51a76-db55-4f0c-b0cb-4e4c526a5bb5"). InnerVolumeSpecName "kube-api-access-qkjbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:58:40.416929 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.416904 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tls-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:40.416929 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.416929 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkjbn\" (UniqueName: \"kubernetes.io/projected/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-kube-api-access-qkjbn\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:40.417089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.416958 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-tmp-dir\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:40.417089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.416968 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5-dshm\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:58:40.810173 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.810137 2571 generic.go:356] "Generic (PLEG): container finished" podID="56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" containerID="2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15" exitCode=0 Aug 04 16:58:40.810353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.810187 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" Aug 04 16:58:40.810353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.810227 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" event={"ID":"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5","Type":"ContainerDied","Data":"2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15"} Aug 04 16:58:40.810353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.810272 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d" event={"ID":"56e51a76-db55-4f0c-b0cb-4e4c526a5bb5","Type":"ContainerDied","Data":"9ad182d8a69d9ed1ad5ad9b16823464a92e89fec694784036af0d424d5c167f6"} Aug 04 16:58:40.810353 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.810290 2571 scope.go:117] "RemoveContainer" containerID="2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15" Aug 04 16:58:40.822403 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.822382 2571 scope.go:117] "RemoveContainer" containerID="2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15" Aug 04 16:58:40.822671 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:58:40.822651 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15\": container with ID starting with 2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15 not found: ID does not exist" containerID="2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15" Aug 04 16:58:40.822722 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.822681 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15"} err="failed to get container status \"2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15\": rpc error: code = NotFound desc = could not find container \"2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15\": container with ID starting with 2631c09891de27962a4ae6628751b8d909c5e746a001d62352b52b6ce0cd5a15 not found: ID does not exist" Aug 04 16:58:40.834220 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.834175 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d"] Aug 04 16:58:40.838047 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:40.838029 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bbe0dc/ipp-route-bbe0dc-kserve-77c44f4c79-9lm4d"] Aug 04 16:58:41.364034 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:41.364000 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" path="/var/lib/kubelet/pods/56e51a76-db55-4f0c-b0cb-4e4c526a5bb5/volumes" Aug 04 16:58:59.419893 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:58:59.419858 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:01.206091 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:01.206055 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:02.195459 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:02.195428 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:07.711106 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:07.711071 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:07.824741 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:07.824704 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:08.019274 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.019148 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9"] Aug 04 16:59:08.019439 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.019417 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" containerID="cri-o://448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65" gracePeriod=30 Aug 04 16:59:08.057608 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.057577 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:08.268016 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.267969 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:08.391512 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391482 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt"] Aug 04 16:59:08.391871 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391860 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" containerName="main" Aug 04 16:59:08.391922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391873 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" containerName="main" Aug 04 16:59:08.391922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391890 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" Aug 04 16:59:08.391922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391895 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" Aug 04 16:59:08.391922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391910 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" Aug 04 16:59:08.391922 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391915 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" Aug 04 16:59:08.392078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391927 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146de1c6-d4aa-4f2c-967f-1b8775635979" containerName="revoke-keys" Aug 04 16:59:08.392078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391933 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="146de1c6-d4aa-4f2c-967f-1b8775635979" containerName="revoke-keys" Aug 04 16:59:08.392078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.391993 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" Aug 04 16:59:08.392078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.392002 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e51a76-db55-4f0c-b0cb-4e4c526a5bb5" containerName="main" Aug 04 16:59:08.392078 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.392010 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="146de1c6-d4aa-4f2c-967f-1b8775635979" containerName="revoke-keys" Aug 04 16:59:08.395102 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.395087 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" Aug 04 16:59:08.397785 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.397766 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:59:08.397879 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.397793 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:59:08.406156 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.406134 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt"] Aug 04 16:59:08.467256 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.467223 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afb7ec5-6b61-4ddf-aabd-948f6b99690a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt\" (UID: \"6afb7ec5-6b61-4ddf-aabd-948f6b99690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" Aug 04 16:59:08.568540 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.568507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afb7ec5-6b61-4ddf-aabd-948f6b99690a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt\" (UID: \"6afb7ec5-6b61-4ddf-aabd-948f6b99690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" Aug 04 16:59:08.569124 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.569106 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afb7ec5-6b61-4ddf-aabd-948f6b99690a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt\" (UID: \"6afb7ec5-6b61-4ddf-aabd-948f6b99690a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" Aug 04 16:59:08.706849 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.706760 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" Aug 04 16:59:08.847836 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.847811 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt"] Aug 04 16:59:08.849652 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:59:08.849617 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afb7ec5_6b61_4ddf_aabd_948f6b99690a.slice/crio-9561f0cf134b9bba1734daa31ded4234ab0868185af52561a3143e33d31c1d95 WatchSource:0}: Error finding container 9561f0cf134b9bba1734daa31ded4234ab0868185af52561a3143e33d31c1d95: Status 404 returned error can't find the container with id 9561f0cf134b9bba1734daa31ded4234ab0868185af52561a3143e33d31c1d95 Aug 04 16:59:08.977923 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:08.977835 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" event={"ID":"6afb7ec5-6b61-4ddf-aabd-948f6b99690a","Type":"ContainerStarted","Data":"9561f0cf134b9bba1734daa31ded4234ab0868185af52561a3143e33d31c1d95"} Aug 04 16:59:09.985363 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:09.985327 2571 generic.go:356] "Generic (PLEG): container finished" podID="6afb7ec5-6b61-4ddf-aabd-948f6b99690a" containerID="48d887159732d0af1223dc0ee90ecd1c2db2e21d2c3134ad60715b4151f583a9" exitCode=0 Aug 04 16:59:09.985805 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:09.985383 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" event={"ID":"6afb7ec5-6b61-4ddf-aabd-948f6b99690a","Type":"ContainerDied","Data":"48d887159732d0af1223dc0ee90ecd1c2db2e21d2c3134ad60715b4151f583a9"} Aug 04 16:59:11.127251 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:11.127221 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" Aug 04 16:59:11.198909 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:11.198879 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afb7ec5-6b61-4ddf-aabd-948f6b99690a-maas-api-service-ca\") pod \"6afb7ec5-6b61-4ddf-aabd-948f6b99690a\" (UID: \"6afb7ec5-6b61-4ddf-aabd-948f6b99690a\") " Aug 04 16:59:11.199283 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:11.199264 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb7ec5-6b61-4ddf-aabd-948f6b99690a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6afb7ec5-6b61-4ddf-aabd-948f6b99690a" (UID: "6afb7ec5-6b61-4ddf-aabd-948f6b99690a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:11.300112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:11.300027 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6afb7ec5-6b61-4ddf-aabd-948f6b99690a-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:11.998659 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:11.998629 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" Aug 04 16:59:11.998840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:11.998669 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt" event={"ID":"6afb7ec5-6b61-4ddf-aabd-948f6b99690a","Type":"ContainerDied","Data":"9561f0cf134b9bba1734daa31ded4234ab0868185af52561a3143e33d31c1d95"} Aug 04 16:59:11.998840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:11.998717 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9561f0cf134b9bba1734daa31ded4234ab0868185af52561a3143e33d31c1d95" Aug 04 16:59:13.019948 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.019907 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:13.284296 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.284226 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:59:13.318429 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318396 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-credential-socket\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318594 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318453 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9wtg\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-kube-api-access-p9wtg\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318594 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318486 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istiod-ca-cert\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318594 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318512 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-socket\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318594 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318539 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-podinfo\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318603 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-envoy\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318637 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-certs\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318703 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:13.318832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318744 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-token\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.318832 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318783 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-data\") pod \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\" (UID: \"f70e46f4-30aa-4baf-bf15-c3513c8e602e\") " Aug 04 16:59:13.319092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.318999 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:13.319092 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.319040 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:13.319223 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.319102 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:13.319223 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.319161 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-credential-socket\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.319223 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.319181 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istiod-ca-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.319223 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.319185 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-data" (OuterVolumeSpecName: "istio-data") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:13.319223 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.319195 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.320970 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.320938 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:13.321322 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.321283 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-kube-api-access-p9wtg" (OuterVolumeSpecName: "kube-api-access-p9wtg") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "kube-api-access-p9wtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:13.321414 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.321294 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-token" (OuterVolumeSpecName: "istio-token") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:13.321414 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.321337 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f70e46f4-30aa-4baf-bf15-c3513c8e602e" (UID: "f70e46f4-30aa-4baf-bf15-c3513c8e602e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 16:59:13.420090 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.420054 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-token\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.420090 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.420084 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-data\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.420327 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.420114 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9wtg\" (UniqueName: \"kubernetes.io/projected/f70e46f4-30aa-4baf-bf15-c3513c8e602e-kube-api-access-p9wtg\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.420327 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.420124 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-workload-socket\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.420327 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.420137 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-podinfo\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:13.420327 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:13.420145 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f70e46f4-30aa-4baf-bf15-c3513c8e602e-istio-envoy\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:14.011763 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.011721 2571 generic.go:356] "Generic (PLEG): container finished" podID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerID="448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65" exitCode=0 Aug 04 16:59:14.011933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.011793 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" Aug 04 16:59:14.011933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.011798 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" event={"ID":"f70e46f4-30aa-4baf-bf15-c3513c8e602e","Type":"ContainerDied","Data":"448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65"} Aug 04 16:59:14.011933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.011854 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9" event={"ID":"f70e46f4-30aa-4baf-bf15-c3513c8e602e","Type":"ContainerDied","Data":"5e1a78d98892ee310e0f3eeb3e8950be66736090522bf1e2c619330340d128ed"} Aug 04 16:59:14.011933 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.011878 2571 scope.go:117] "RemoveContainer" containerID="448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65" Aug 04 16:59:14.021396 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.021375 2571 scope.go:117] "RemoveContainer" containerID="4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22" Aug 04 16:59:14.029612 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.029594 2571 scope.go:117] "RemoveContainer" containerID="448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65" Aug 04 16:59:14.029858 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:59:14.029838 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65\": container with ID starting with 448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65 not found: ID does not exist" containerID="448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65" Aug 04 16:59:14.029917 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.029867 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65"} err="failed to get container status \"448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65\": rpc error: code = NotFound desc = could not find container \"448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65\": container with ID starting with 448aa2c5aa4006c3da73a611c352f8e757bcd51102d853e7549f1ae0df7f8a65 not found: ID does not exist" Aug 04 16:59:14.029917 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.029883 2571 scope.go:117] "RemoveContainer" containerID="4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22" Aug 04 16:59:14.030147 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:59:14.030128 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22\": container with ID starting with 4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22 not found: ID does not exist" containerID="4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22" Aug 04 16:59:14.030231 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.030151 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22"} err="failed to get container status \"4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22\": rpc error: code = NotFound desc = could not find container \"4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22\": container with ID starting with 4527e648f1a81f33645ccfd627cadc6ad0ea6b83965a8d00daebc03beedb5e22 not found: ID does not exist" Aug 04 16:59:14.031629 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.031611 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9"] Aug 04 16:59:14.035161 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:14.035142 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bbe0dc-openshift-default-74fdb9774b-98kc9"] Aug 04 16:59:15.362975 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:15.362942 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" path="/var/lib/kubelet/pods/f70e46f4-30aa-4baf-bf15-c3513c8e602e/volumes" Aug 04 16:59:18.824553 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:18.824515 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:18.924499 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:18.924360 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:19.003670 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:19.003628 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt"] Aug 04 16:59:19.005370 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:19.005343 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:19.005992 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:19.005973 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-54f057-x66kt"] Aug 04 16:59:19.136345 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:19.136235 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:19.363891 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:19.363856 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afb7ec5-6b61-4ddf-aabd-948f6b99690a" path="/var/lib/kubelet/pods/6afb7ec5-6b61-4ddf-aabd-948f6b99690a/volumes" Aug 04 16:59:19.406321 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:19.406229 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:21.378734 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:21.378706 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:59:25.077026 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.076951 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:25.237693 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.237664 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:25.348137 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.348043 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j"] Aug 04 16:59:25.348340 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.348313 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" containerID="cri-o://d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88" gracePeriod=30 Aug 04 16:59:25.363763 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.363730 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:25.592021 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.591984 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:25.742552 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.742472 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd"] Aug 04 16:59:25.742939 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.742921 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" Aug 04 16:59:25.742991 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.742941 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" Aug 04 16:59:25.742991 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.742969 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb7ec5-6b61-4ddf-aabd-948f6b99690a" containerName="revoke-keys" Aug 04 16:59:25.742991 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.742975 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb7ec5-6b61-4ddf-aabd-948f6b99690a" containerName="revoke-keys" Aug 04 16:59:25.743112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.743047 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" Aug 04 16:59:25.743112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.743055 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e9deff-9bc1-47d7-9841-1d5a196da819" containerName="istio-proxy" Aug 04 16:59:25.743112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.743061 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afb7ec5-6b61-4ddf-aabd-948f6b99690a" containerName="revoke-keys" Aug 04 16:59:25.743112 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.743070 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" Aug 04 16:59:25.746043 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.746027 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" Aug 04 16:59:25.748696 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.748674 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:59:25.748812 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.748674 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:59:25.753016 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.752995 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd"] Aug 04 16:59:25.831830 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.831795 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ae97a-f869-4327-84ac-d80cd19d5371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-90d707-prrxd\" (UID: \"b10ae97a-f869-4327-84ac-d80cd19d5371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" Aug 04 16:59:25.933265 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.933233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ae97a-f869-4327-84ac-d80cd19d5371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-90d707-prrxd\" (UID: \"b10ae97a-f869-4327-84ac-d80cd19d5371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" Aug 04 16:59:25.933879 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:25.933860 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ae97a-f869-4327-84ac-d80cd19d5371-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-90d707-prrxd\" (UID: \"b10ae97a-f869-4327-84ac-d80cd19d5371\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" Aug 04 16:59:26.058299 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:26.058272 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" Aug 04 16:59:26.402614 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:26.402571 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd"] Aug 04 16:59:26.403704 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:59:26.403669 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10ae97a_f869_4327_84ac_d80cd19d5371.slice/crio-2d75cbbf274121527a80dc50b8116c36bc3d730617effe7700c82f0886775061 WatchSource:0}: Error finding container 2d75cbbf274121527a80dc50b8116c36bc3d730617effe7700c82f0886775061: Status 404 returned error can't find the container with id 2d75cbbf274121527a80dc50b8116c36bc3d730617effe7700c82f0886775061 Aug 04 16:59:27.098019 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:27.097984 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" event={"ID":"b10ae97a-f869-4327-84ac-d80cd19d5371","Type":"ContainerStarted","Data":"5d16e7c551e061d1a878fdc74af898baa166eba8cbe8d034ffe5af09dc08ce44"} Aug 04 16:59:27.098238 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:27.098075 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" event={"ID":"b10ae97a-f869-4327-84ac-d80cd19d5371","Type":"ContainerStarted","Data":"2d75cbbf274121527a80dc50b8116c36bc3d730617effe7700c82f0886775061"} Aug 04 16:59:27.117398 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:27.117355 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" podStartSLOduration=2.117341056 podStartE2EDuration="2.117341056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:59:27.116341695 +0000 UTC m=+3776.209941938" watchObservedRunningTime="2026-08-04 16:59:27.117341056 +0000 UTC m=+3776.210941304" Aug 04 16:59:28.105673 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:28.105634 2571 generic.go:356] "Generic (PLEG): container finished" podID="b10ae97a-f869-4327-84ac-d80cd19d5371" containerID="5d16e7c551e061d1a878fdc74af898baa166eba8cbe8d034ffe5af09dc08ce44" exitCode=0 Aug 04 16:59:28.106065 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:28.105730 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" event={"ID":"b10ae97a-f869-4327-84ac-d80cd19d5371","Type":"ContainerDied","Data":"5d16e7c551e061d1a878fdc74af898baa166eba8cbe8d034ffe5af09dc08ce44"} Aug 04 16:59:29.242138 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:29.242114 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" Aug 04 16:59:29.364282 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:29.364190 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ae97a-f869-4327-84ac-d80cd19d5371-maas-api-service-ca\") pod \"b10ae97a-f869-4327-84ac-d80cd19d5371\" (UID: \"b10ae97a-f869-4327-84ac-d80cd19d5371\") " Aug 04 16:59:29.364500 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:29.364479 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10ae97a-f869-4327-84ac-d80cd19d5371-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b10ae97a-f869-4327-84ac-d80cd19d5371" (UID: "b10ae97a-f869-4327-84ac-d80cd19d5371"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:29.465007 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:29.464981 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10ae97a-f869-4327-84ac-d80cd19d5371-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.118328 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.118294 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" event={"ID":"b10ae97a-f869-4327-84ac-d80cd19d5371","Type":"ContainerDied","Data":"2d75cbbf274121527a80dc50b8116c36bc3d730617effe7700c82f0886775061"} Aug 04 16:59:30.118504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.118348 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d75cbbf274121527a80dc50b8116c36bc3d730617effe7700c82f0886775061" Aug 04 16:59:30.118504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.118304 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd" Aug 04 16:59:30.599370 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.599345 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:59:30.673151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673071 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9082291c-c94c-4214-bbd8-c2edab5b56c6-istiod-ca-cert\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673109 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-envoy\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673151 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673150 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-podinfo\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673431 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673179 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-credential-socket\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673431 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673198 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-data\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673431 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673239 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfsgt\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-kube-api-access-hfsgt\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673431 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673270 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-socket\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673431 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673336 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-certs\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673431 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673397 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-token\") pod \"9082291c-c94c-4214-bbd8-c2edab5b56c6\" (UID: \"9082291c-c94c-4214-bbd8-c2edab5b56c6\") " Aug 04 16:59:30.673742 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673598 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:30.673742 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673514 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9082291c-c94c-4214-bbd8-c2edab5b56c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:30.673742 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673619 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:30.673907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673804 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:30.673907 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673816 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:30.674024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.673980 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9082291c-c94c-4214-bbd8-c2edab5b56c6-istiod-ca-cert\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.674024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.674000 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-credential-socket\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.674024 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.674016 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-data\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.674182 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.674026 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-socket\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.674182 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.674034 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-workload-certs\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.675376 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.675350 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:59:30.675489 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.675470 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:30.675892 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.675865 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Aug 04 16:59:30.675892 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.675875 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-kube-api-access-hfsgt" (OuterVolumeSpecName: "kube-api-access-hfsgt") pod "9082291c-c94c-4214-bbd8-c2edab5b56c6" (UID: "9082291c-c94c-4214-bbd8-c2edab5b56c6"). InnerVolumeSpecName "kube-api-access-hfsgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:59:30.775238 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.775185 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-podinfo\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.775379 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.775245 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfsgt\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-kube-api-access-hfsgt\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.775379 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.775263 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-token\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:30.775379 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:30.775275 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9082291c-c94c-4214-bbd8-c2edab5b56c6-istio-envoy\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:31.127326 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.127293 2571 generic.go:356] "Generic (PLEG): container finished" podID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerID="d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88" exitCode=0 Aug 04 16:59:31.127499 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.127333 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" event={"ID":"9082291c-c94c-4214-bbd8-c2edab5b56c6","Type":"ContainerDied","Data":"d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88"} Aug 04 16:59:31.127499 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.127350 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" Aug 04 16:59:31.127499 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.127390 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j" event={"ID":"9082291c-c94c-4214-bbd8-c2edab5b56c6","Type":"ContainerDied","Data":"9ce0a3fe06bd1cb8264f0523b82460bf87d7386ec02c8465ca66b68f58281a61"} Aug 04 16:59:31.127499 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.127423 2571 scope.go:117] "RemoveContainer" containerID="d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88" Aug 04 16:59:31.137148 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.137130 2571 scope.go:117] "RemoveContainer" containerID="bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326" Aug 04 16:59:31.145647 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.145629 2571 scope.go:117] "RemoveContainer" containerID="d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88" Aug 04 16:59:31.145882 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:59:31.145866 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88\": container with ID starting with d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88 not found: ID does not exist" containerID="d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88" Aug 04 16:59:31.145932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.145888 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88"} err="failed to get container status \"d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88\": rpc error: code = NotFound desc = could not find container \"d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88\": container with ID starting with d3973b288307e956e1731f7cc0adeccba8bfdbdb3264ff2d957bfb93efdf3b88 not found: ID does not exist" Aug 04 16:59:31.145932 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.145903 2571 scope.go:117] "RemoveContainer" containerID="bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326" Aug 04 16:59:31.146122 ip-10-0-137-14 kubenswrapper[2571]: E0804 16:59:31.146104 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326\": container with ID starting with bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326 not found: ID does not exist" containerID="bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326" Aug 04 16:59:31.146168 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.146129 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326"} err="failed to get container status \"bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326\": rpc error: code = NotFound desc = could not find container \"bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326\": container with ID starting with bc34ab34faa997a58f0c4e78b34449d3396da064dc33c754abff66153f20d326 not found: ID does not exist" Aug 04 16:59:31.150089 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.150058 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j"] Aug 04 16:59:31.155454 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.155433 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-54f057-openshift-default-898995c54-cn76j"] Aug 04 16:59:31.363060 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:31.363032 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" path="/var/lib/kubelet/pods/9082291c-c94c-4214-bbd8-c2edab5b56c6/volumes" Aug 04 16:59:36.302615 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:36.302573 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:36.376365 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:36.376330 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:36.442505 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:36.442466 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:36.487738 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:36.487704 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd"] Aug 04 16:59:36.492672 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:36.492649 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-90d707-prrxd"] Aug 04 16:59:36.510958 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:36.510931 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:36.879034 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:36.879002 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:37.363423 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:37.363384 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b10ae97a-f869-4327-84ac-d80cd19d5371" path="/var/lib/kubelet/pods/b10ae97a-f869-4327-84ac-d80cd19d5371/volumes" Aug 04 16:59:38.485504 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:38.485465 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:59:43.018354 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.018316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:43.119476 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.119438 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:43.304574 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.304538 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:43.532364 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.532316 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:43.724372 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724280 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f"] Aug 04 16:59:43.724757 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724739 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" Aug 04 16:59:43.724840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724760 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" Aug 04 16:59:43.724840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724776 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" Aug 04 16:59:43.724840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724785 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70e46f4-30aa-4baf-bf15-c3513c8e602e" containerName="istio-proxy" Aug 04 16:59:43.724840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724825 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b10ae97a-f869-4327-84ac-d80cd19d5371" containerName="revoke-keys" Aug 04 16:59:43.724840 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724833 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b10ae97a-f869-4327-84ac-d80cd19d5371" containerName="revoke-keys" Aug 04 16:59:43.725083 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724931 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" Aug 04 16:59:43.725083 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724945 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" Aug 04 16:59:43.725083 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.724958 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b10ae97a-f869-4327-84ac-d80cd19d5371" containerName="revoke-keys" Aug 04 16:59:43.729675 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.729655 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" Aug 04 16:59:43.734014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.733552 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 16:59:43.734014 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.733813 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 16:59:43.735804 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.735775 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f"] Aug 04 16:59:43.793921 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.793895 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2be5901-53ee-45c1-b63b-7922ccd028ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f\" (UID: \"a2be5901-53ee-45c1-b63b-7922ccd028ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" Aug 04 16:59:43.894753 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.894721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2be5901-53ee-45c1-b63b-7922ccd028ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f\" (UID: \"a2be5901-53ee-45c1-b63b-7922ccd028ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" Aug 04 16:59:43.895331 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:43.895312 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2be5901-53ee-45c1-b63b-7922ccd028ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f\" (UID: \"a2be5901-53ee-45c1-b63b-7922ccd028ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" Aug 04 16:59:44.041423 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:44.041352 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" Aug 04 16:59:44.178957 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:44.178933 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f"] Aug 04 16:59:44.180631 ip-10-0-137-14 kubenswrapper[2571]: W0804 16:59:44.180604 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2be5901_53ee_45c1_b63b_7922ccd028ac.slice/crio-98b602eef83f8157de9987cd9f1d4770bbd85fe3dff2014b4c5b7e2095ac1c4a WatchSource:0}: Error finding container 98b602eef83f8157de9987cd9f1d4770bbd85fe3dff2014b4c5b7e2095ac1c4a: Status 404 returned error can't find the container with id 98b602eef83f8157de9987cd9f1d4770bbd85fe3dff2014b4c5b7e2095ac1c4a Aug 04 16:59:44.210369 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:44.210342 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" event={"ID":"a2be5901-53ee-45c1-b63b-7922ccd028ac","Type":"ContainerStarted","Data":"98b602eef83f8157de9987cd9f1d4770bbd85fe3dff2014b4c5b7e2095ac1c4a"} Aug 04 16:59:45.218508 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:45.218421 2571 generic.go:356] "Generic (PLEG): container finished" podID="a2be5901-53ee-45c1-b63b-7922ccd028ac" containerID="fb789be360a4754eb44c8157a0040fc78ff046464e242c79505ee901e7217b14" exitCode=0 Aug 04 16:59:45.218508 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:45.218469 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" event={"ID":"a2be5901-53ee-45c1-b63b-7922ccd028ac","Type":"ContainerDied","Data":"fb789be360a4754eb44c8157a0040fc78ff046464e242c79505ee901e7217b14"} Aug 04 16:59:46.360273 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:46.360250 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" Aug 04 16:59:46.418802 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:46.418777 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2be5901-53ee-45c1-b63b-7922ccd028ac-maas-api-service-ca\") pod \"a2be5901-53ee-45c1-b63b-7922ccd028ac\" (UID: \"a2be5901-53ee-45c1-b63b-7922ccd028ac\") " Aug 04 16:59:46.419096 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:46.419075 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2be5901-53ee-45c1-b63b-7922ccd028ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2be5901-53ee-45c1-b63b-7922ccd028ac" (UID: "a2be5901-53ee-45c1-b63b-7922ccd028ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:59:46.519942 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:46.519879 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2be5901-53ee-45c1-b63b-7922ccd028ac-maas-api-service-ca\") on node \"ip-10-0-137-14.ec2.internal\" DevicePath \"\"" Aug 04 16:59:47.231354 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:47.231314 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" event={"ID":"a2be5901-53ee-45c1-b63b-7922ccd028ac","Type":"ContainerDied","Data":"98b602eef83f8157de9987cd9f1d4770bbd85fe3dff2014b4c5b7e2095ac1c4a"} Aug 04 16:59:47.231522 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:47.231371 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98b602eef83f8157de9987cd9f1d4770bbd85fe3dff2014b4c5b7e2095ac1c4a" Aug 04 16:59:47.231522 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:47.231327 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f" Aug 04 16:59:48.923369 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:48.923329 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:48.982828 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:48.982790 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:49.046187 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:49.046150 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:49.192951 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:49.192868 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f"] Aug 04 16:59:49.195237 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:49.195213 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-5a3cb8-p5r2f"] Aug 04 16:59:49.270967 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:49.270933 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:49.363822 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:49.363792 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2be5901-53ee-45c1-b63b-7922ccd028ac" path="/var/lib/kubelet/pods/a2be5901-53ee-45c1-b63b-7922ccd028ac/volumes" Aug 04 16:59:49.487266 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:49.487180 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:51.084439 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:51.084404 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-5h7sv"] Aug 04 16:59:55.694154 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:55.694120 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:55.758662 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:55.758622 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:55.961565 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:55.961477 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 16:59:56.253458 ip-10-0-137-14 kubenswrapper[2571]: I0804 16:59:56.253364 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 17:00:00.127656 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.127625 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7"] Aug 04 17:00:00.128044 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.128015 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2be5901-53ee-45c1-b63b-7922ccd028ac" containerName="revoke-keys" Aug 04 17:00:00.128044 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.128026 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2be5901-53ee-45c1-b63b-7922ccd028ac" containerName="revoke-keys" Aug 04 17:00:00.128129 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.128048 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" Aug 04 17:00:00.128129 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.128054 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9082291c-c94c-4214-bbd8-c2edab5b56c6" containerName="istio-proxy" Aug 04 17:00:00.128129 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.128117 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2be5901-53ee-45c1-b63b-7922ccd028ac" containerName="revoke-keys" Aug 04 17:00:00.131347 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.131331 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7" Aug 04 17:00:00.133726 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.133704 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-psz5k\"" Aug 04 17:00:00.138517 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.138492 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7"] Aug 04 17:00:00.143857 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.143834 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7" Aug 04 17:00:00.290985 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.290959 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7"] Aug 04 17:00:00.293327 ip-10-0-137-14 kubenswrapper[2571]: W0804 17:00:00.293296 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdead936_24df_4247_83b6_4248dfa47703.slice/crio-1c759a66a9d4885be27ffb26c1eb2e4031d6de0f605dedc6bfc0852f2231d1d1 WatchSource:0}: Error finding container 1c759a66a9d4885be27ffb26c1eb2e4031d6de0f605dedc6bfc0852f2231d1d1: Status 404 returned error can't find the container with id 1c759a66a9d4885be27ffb26c1eb2e4031d6de0f605dedc6bfc0852f2231d1d1 Aug 04 17:00:00.303497 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:00.303471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7" event={"ID":"bdead936-24df-4247-83b6-4248dfa47703","Type":"ContainerStarted","Data":"1c759a66a9d4885be27ffb26c1eb2e4031d6de0f605dedc6bfc0852f2231d1d1"} Aug 04 17:00:01.311039 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:01.311010 2571 generic.go:356] "Generic (PLEG): container finished" podID="bdead936-24df-4247-83b6-4248dfa47703" containerID="981ecc9a8b64d356631921e34277294a6b4d20167872c73e40bab6d729c9ba13" exitCode=0 Aug 04 17:00:01.311465 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:01.311080 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7" event={"ID":"bdead936-24df-4247-83b6-4248dfa47703","Type":"ContainerDied","Data":"981ecc9a8b64d356631921e34277294a6b4d20167872c73e40bab6d729c9ba13"} Aug 04 17:00:02.449236 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:02.449195 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7" Aug 04 17:00:02.488278 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:02.488250 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kv24b_6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef/manager/0.log" Aug 04 17:00:02.727786 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:02.727698 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-b6jwv_b76d46f0-0b35-43bd-b210-353a1f617907/server/0.log" Aug 04 17:00:02.849325 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:02.849279 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v6xnk_c2b7dc31-4af2-4db3-afd3-4fd6b644084f/manager/0.log" Aug 04 17:00:03.242680 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:03.242652 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-sps7q_bbd0d08e-cbbd-4937-9858-a698d11eed1e/manager/0.log" Aug 04 17:00:03.258272 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:03.258244 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-sps7q_bbd0d08e-cbbd-4937-9858-a698d11eed1e/copy-manifests/0.log" Aug 04 17:00:03.322633 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:03.322608 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7" Aug 04 17:00:03.322784 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:03.322642 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29764380-dr4b7" event={"ID":"bdead936-24df-4247-83b6-4248dfa47703","Type":"ContainerDied","Data":"1c759a66a9d4885be27ffb26c1eb2e4031d6de0f605dedc6bfc0852f2231d1d1"} Aug 04 17:00:03.322784 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:03.322687 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c759a66a9d4885be27ffb26c1eb2e4031d6de0f605dedc6bfc0852f2231d1d1" Aug 04 17:00:03.341075 ip-10-0-137-14 kubenswrapper[2571]: E0804 17:00:03.341051 2571 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdead936_24df_4247_83b6_4248dfa47703.slice\": RecentStats: unable to find data in memory cache]" Aug 04 17:00:03.981016 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:03.980987 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764365-hp6gt_af21625c-f73d-484b-b029-e36b0a1849ea/cleanup/0.log" Aug 04 17:00:04.111590 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:04.111565 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29764380-dr4b7_bdead936-24df-4247-83b6-4248dfa47703/cleanup/0.log" Aug 04 17:00:05.207741 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:05.207708 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-c8jf6_60059be3-f43e-4806-a420-c34245f4fc23/manager/0.log" Aug 04 17:00:05.555233 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:05.555193 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-m2ksb_90973b46-253c-4e53-aac8-ed789fd9c0dd/manager/0.log" Aug 04 17:00:05.675500 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:05.675472 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-5h7sv_a94ae9e4-8d3d-45fc-96ec-648c2690c45e/limitador/0.log" Aug 04 17:00:06.290138 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:06.290106 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rjgkk_4380e24f-b3fc-4d52-a094-c4de038657fe/discovery/0.log" Aug 04 17:00:06.400929 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:06.400893 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-kx7qj_afa294db-fb48-47cd-84a8-2260c93aca2d/kube-auth-proxy/0.log" Aug 04 17:00:07.806148 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:07.806121 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-rsxkw_c87e1f70-ca9d-4586-8b89-fe132171c3f7/main/0.log" Aug 04 17:00:14.059524 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:14.059486 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 17:00:14.163027 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:14.162993 2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" limit=1024 Aug 04 17:00:16.878671 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:16.878643 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dknzx_818a2360-3c98-4f8d-b540-d1c90b7d1113/global-pull-secret-syncer/0.log" Aug 04 17:00:16.962141 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:16.962114 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8f7lc_03867d85-2528-4dfa-9618-8e8d14aeca1a/konnectivity-agent/0.log" Aug 04 17:00:17.094269 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:17.094236 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-14.ec2.internal_2b2c461907233d4b90cae1267fdce864/haproxy/0.log" Aug 04 17:00:22.902304 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:22.902273 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-c8jf6_60059be3-f43e-4806-a420-c34245f4fc23/manager/0.log" Aug 04 17:00:23.039586 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:23.039556 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-m2ksb_90973b46-253c-4e53-aac8-ed789fd9c0dd/manager/0.log" Aug 04 17:00:23.061135 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:23.061109 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-5h7sv_a94ae9e4-8d3d-45fc-96ec-648c2690c45e/limitador/0.log" Aug 04 17:00:24.664339 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.664314 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_006a0e30-830b-422d-926b-1a8a51842d8f/alertmanager/0.log" Aug 04 17:00:24.692663 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.692640 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_006a0e30-830b-422d-926b-1a8a51842d8f/config-reloader/0.log" Aug 04 17:00:24.714718 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.714701 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_006a0e30-830b-422d-926b-1a8a51842d8f/kube-rbac-proxy-web/0.log" Aug 04 17:00:24.735898 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.735874 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_006a0e30-830b-422d-926b-1a8a51842d8f/kube-rbac-proxy/0.log" Aug 04 17:00:24.760054 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.760030 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_006a0e30-830b-422d-926b-1a8a51842d8f/kube-rbac-proxy-metric/0.log" Aug 04 17:00:24.781611 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.781592 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_006a0e30-830b-422d-926b-1a8a51842d8f/prom-label-proxy/0.log" Aug 04 17:00:24.807007 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.806986 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_006a0e30-830b-422d-926b-1a8a51842d8f/init-config-reloader/0.log" Aug 04 17:00:24.848952 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.848926 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-jmlx5_79dae8ed-e996-4bcb-bdfb-862526c96a37/cluster-monitoring-operator/0.log" Aug 04 17:00:24.965623 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.965549 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-697h2_73ee198a-29e5-43f2-80b4-92e28c7cc660/monitoring-plugin/0.log" Aug 04 17:00:24.994660 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:24.994636 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hnkg_e6b15423-4ffc-4055-9c27-68928094fff7/node-exporter/0.log" Aug 04 17:00:25.014765 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.014742 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hnkg_e6b15423-4ffc-4055-9c27-68928094fff7/kube-rbac-proxy/0.log" Aug 04 17:00:25.034780 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.034757 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hnkg_e6b15423-4ffc-4055-9c27-68928094fff7/init-textfile/0.log" Aug 04 17:00:25.246005 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.245833 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-lbn9s_aaa34a74-128c-489d-9427-eda3783de4bd/kube-rbac-proxy-main/0.log" Aug 04 17:00:25.266600 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.266575 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-lbn9s_aaa34a74-128c-489d-9427-eda3783de4bd/kube-rbac-proxy-self/0.log" Aug 04 17:00:25.289658 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.289633 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6bb4d7b4dd-lbn9s_aaa34a74-128c-489d-9427-eda3783de4bd/openshift-state-metrics/0.log" Aug 04 17:00:25.559679 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.559651 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-66685ff6d4-kps44_054e280d-3429-450e-87eb-0793502df9d5/prometheus-operator-admission-webhook/0.log" Aug 04 17:00:25.603549 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.603524 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cb56c5f9c-c6khc_b692cf55-5d89-4447-81a9-e4f426962d47/telemeter-client/0.log" Aug 04 17:00:25.624791 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.624770 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cb56c5f9c-c6khc_b692cf55-5d89-4447-81a9-e4f426962d47/reload/0.log" Aug 04 17:00:25.644843 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:25.644802 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cb56c5f9c-c6khc_b692cf55-5d89-4447-81a9-e4f426962d47/kube-rbac-proxy/0.log" Aug 04 17:00:28.042908 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.042876 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bc8d79ddd-l9hm4_3d79a012-9886-4c90-8286-f885ae0adcab/console/0.log" Aug 04 17:00:28.114801 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.114753 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h"] Aug 04 17:00:28.115219 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.115187 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdead936-24df-4247-83b6-4248dfa47703" containerName="cleanup" Aug 04 17:00:28.115219 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.115216 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdead936-24df-4247-83b6-4248dfa47703" containerName="cleanup" Aug 04 17:00:28.115309 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.115284 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdead936-24df-4247-83b6-4248dfa47703" containerName="cleanup" Aug 04 17:00:28.118479 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.118463 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.120945 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.120925 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2bcd\"/\"kube-root-ca.crt\"" Aug 04 17:00:28.121125 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.120980 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b2bcd\"/\"default-dockercfg-5lghf\"" Aug 04 17:00:28.122007 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.121989 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b2bcd\"/\"openshift-service-ca.crt\"" Aug 04 17:00:28.127417 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.127395 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h"] Aug 04 17:00:28.284493 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.284459 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-lib-modules\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.284652 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.284537 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgvh\" (UniqueName: \"kubernetes.io/projected/81278c54-4707-4d9f-9b1f-55111f62ea87-kube-api-access-7tgvh\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.284652 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.284577 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-sys\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.284729 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.284661 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-podres\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.284729 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.284701 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-proc\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385310 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385278 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-podres\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385473 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385319 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-proc\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385473 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385352 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-lib-modules\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385473 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385414 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-proc\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385473 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgvh\" (UniqueName: \"kubernetes.io/projected/81278c54-4707-4d9f-9b1f-55111f62ea87-kube-api-access-7tgvh\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385473 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385455 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-podres\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385707 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385499 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-sys\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385707 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385511 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-lib-modules\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.385707 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.385595 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81278c54-4707-4d9f-9b1f-55111f62ea87-sys\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.393622 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.393595 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgvh\" (UniqueName: \"kubernetes.io/projected/81278c54-4707-4d9f-9b1f-55111f62ea87-kube-api-access-7tgvh\") pod \"perf-node-gather-daemonset-mfc4h\" (UID: \"81278c54-4707-4d9f-9b1f-55111f62ea87\") " pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.429628 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.429601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:28.561126 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:28.561097 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h"] Aug 04 17:00:28.562511 ip-10-0-137-14 kubenswrapper[2571]: W0804 17:00:28.562485 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod81278c54_4707_4d9f_9b1f_55111f62ea87.slice/crio-cf9c2127136157c330dffb7b213cf1aaae8a84ae5aa0512e52e03ba2a2401222 WatchSource:0}: Error finding container cf9c2127136157c330dffb7b213cf1aaae8a84ae5aa0512e52e03ba2a2401222: Status 404 returned error can't find the container with id cf9c2127136157c330dffb7b213cf1aaae8a84ae5aa0512e52e03ba2a2401222 Aug 04 17:00:29.467867 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:29.467826 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" event={"ID":"81278c54-4707-4d9f-9b1f-55111f62ea87","Type":"ContainerStarted","Data":"b0e7ae9b66438d1d56472ae17a6320578f70035677c9e20f2e00ce04a091a9a6"} Aug 04 17:00:29.468281 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:29.467888 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" event={"ID":"81278c54-4707-4d9f-9b1f-55111f62ea87","Type":"ContainerStarted","Data":"cf9c2127136157c330dffb7b213cf1aaae8a84ae5aa0512e52e03ba2a2401222"} Aug 04 17:00:29.468281 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:29.467920 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:29.484998 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:29.484954 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" podStartSLOduration=1.484937984 podStartE2EDuration="1.484937984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:00:29.483896246 +0000 UTC m=+3838.577496495" watchObservedRunningTime="2026-08-04 17:00:29.484937984 +0000 UTC m=+3838.578538278" Aug 04 17:00:29.930560 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:29.930531 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbxx9_955984cd-2260-47fe-a83e-1986fa17419a/dns/0.log" Aug 04 17:00:29.949434 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:29.949409 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbxx9_955984cd-2260-47fe-a83e-1986fa17419a/kube-rbac-proxy/0.log" Aug 04 17:00:29.992588 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:29.992563 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vbw9c_ba50246f-15ba-4589-9ae4-f38e8169ceec/dns-node-resolver/0.log" Aug 04 17:00:30.548137 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:30.548105 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69558cbb7-9j4hh_bd632101-8637-4b0d-a51f-a3277af3b58e/registry/0.log" Aug 04 17:00:30.596660 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:30.596631 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7pxh8_e97926c6-57de-4b89-9fdb-fe0d7310fb68/node-ca/0.log" Aug 04 17:00:31.740427 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:31.740388 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rjgkk_4380e24f-b3fc-4d52-a094-c4de038657fe/discovery/0.log" Aug 04 17:00:31.764415 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:31.764388 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-kx7qj_afa294db-fb48-47cd-84a8-2260c93aca2d/kube-auth-proxy/0.log" Aug 04 17:00:32.706950 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:32.706923 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-j94lw_fc13c2fb-981d-4308-8680-a1d97241dbee/serve-healthcheck-canary/0.log" Aug 04 17:00:33.232048 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:33.232021 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-blmkg_5955abed-5484-4f74-a6ff-8490cc3114b3/insights-operator/0.log" Aug 04 17:00:33.233453 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:33.233431 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-blmkg_5955abed-5484-4f74-a6ff-8490cc3114b3/insights-operator/1.log" Aug 04 17:00:33.392249 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:33.392184 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdzkk_3ce2d520-0b80-43f4-abac-1e74c4cb7af5/kube-rbac-proxy/0.log" Aug 04 17:00:33.412519 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:33.412493 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdzkk_3ce2d520-0b80-43f4-abac-1e74c4cb7af5/exporter/0.log" Aug 04 17:00:33.432423 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:33.432404 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jdzkk_3ce2d520-0b80-43f4-abac-1e74c4cb7af5/extractor/0.log" Aug 04 17:00:35.483581 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:35.483552 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b2bcd/perf-node-gather-daemonset-mfc4h" Aug 04 17:00:35.783654 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:35.783569 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kv24b_6b0e5f25-ce3f-4b16-8462-dbeb6773a6ef/manager/0.log" Aug 04 17:00:36.031384 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:36.031353 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-b6jwv_b76d46f0-0b35-43bd-b210-353a1f617907/server/0.log" Aug 04 17:00:36.105839 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:36.105814 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-v6xnk_c2b7dc31-4af2-4db3-afd3-4fd6b644084f/manager/0.log" Aug 04 17:00:36.389131 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:36.389055 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-sps7q_bbd0d08e-cbbd-4937-9858-a698d11eed1e/manager/0.log" Aug 04 17:00:36.408655 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:36.408636 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dfb598967-sps7q_bbd0d08e-cbbd-4937-9858-a698d11eed1e/copy-manifests/0.log" Aug 04 17:00:37.778472 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:37.778441 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rjgkk_4380e24f-b3fc-4d52-a094-c4de038657fe/discovery/0.log" Aug 04 17:00:37.789041 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:37.789020 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-kx7qj_afa294db-fb48-47cd-84a8-2260c93aca2d/kube-auth-proxy/0.log" Aug 04 17:00:37.844478 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:37.844453 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nx2ch_a67e72cf-b697-40b3-a754-11e251b5999e/openshift-lws-operator/0.log" Aug 04 17:00:39.496964 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:39.496943 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-rjgkk_4380e24f-b3fc-4d52-a094-c4de038657fe/discovery/0.log" Aug 04 17:00:39.508029 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:39.508009 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78b5c5f4c4-kx7qj_afa294db-fb48-47cd-84a8-2260c93aca2d/kube-auth-proxy/0.log" Aug 04 17:00:43.205924 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:43.205897 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-w2qqd_6f542629-2dd8-4570-ad36-652186645e9e/kube-storage-version-migrator-operator/1.log" Aug 04 17:00:43.206628 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:43.206611 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-w2qqd_6f542629-2dd8-4570-ad36-652186645e9e/kube-storage-version-migrator-operator/0.log" Aug 04 17:00:44.199655 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.199627 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zhbq_a32c5e48-78e8-4816-80c3-2d38d50a359f/kube-multus-additional-cni-plugins/0.log" Aug 04 17:00:44.219874 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.219842 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zhbq_a32c5e48-78e8-4816-80c3-2d38d50a359f/egress-router-binary-copy/0.log" Aug 04 17:00:44.239253 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.239194 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zhbq_a32c5e48-78e8-4816-80c3-2d38d50a359f/cni-plugins/0.log" Aug 04 17:00:44.258853 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.258828 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zhbq_a32c5e48-78e8-4816-80c3-2d38d50a359f/bond-cni-plugin/0.log" Aug 04 17:00:44.278118 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.278088 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zhbq_a32c5e48-78e8-4816-80c3-2d38d50a359f/routeoverride-cni/0.log" Aug 04 17:00:44.300212 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.300185 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zhbq_a32c5e48-78e8-4816-80c3-2d38d50a359f/whereabouts-cni-bincopy/0.log" Aug 04 17:00:44.319435 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.319414 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6zhbq_a32c5e48-78e8-4816-80c3-2d38d50a359f/whereabouts-cni/0.log" Aug 04 17:00:44.678321 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.678288 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-swj4m_145a5d49-25db-4674-838c-c2ce7612deb8/kube-multus/0.log" Aug 04 17:00:44.735145 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.735119 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ltd4v_d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d/network-metrics-daemon/0.log" Aug 04 17:00:44.752638 ip-10-0-137-14 kubenswrapper[2571]: I0804 17:00:44.752615 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ltd4v_d6dd1c6f-ee03-4a25-b3cb-7187d9e2d78d/kube-rbac-proxy/0.log"