Jul 24 17:03:02.380762 ip-10-0-138-185 systemd[1]: Starting Kubernetes Kubelet... Jul 24 17:03:02.785286 ip-10-0-138-185 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:03:02.785286 ip-10-0-138-185 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 17:03:02.785286 ip-10-0-138-185 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:03:02.785286 ip-10-0-138-185 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 17:03:02.785286 ip-10-0-138-185 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:03:02.786063 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.785981 2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 17:03:02.789534 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789514 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:03:02.789534 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789531 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:03:02.789534 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789537 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:03:02.789534 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789541 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789546 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789550 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789553 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789557 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789562 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789566 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789569 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789573 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789577 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789581 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789584 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789588 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789592 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789596 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789600 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789605 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789609 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789613 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:03:02.789784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789619 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789626 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789630 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789634 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789638 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789642 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789645 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789649 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789654 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789658 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789662 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789666 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789677 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789682 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789686 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789692 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789696 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789700 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789704 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:03:02.790398 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789711 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789718 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789723 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789727 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789732 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789737 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789741 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789745 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789749 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789753 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789758 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789762 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789767 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789772 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789776 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789780 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789784 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789788 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789793 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789797 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:03:02.790892 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789801 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789805 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789809 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789813 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789818 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789822 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789827 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789831 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789836 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789841 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789846 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789850 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789854 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789858 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789864 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789868 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789872 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789876 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789881 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789889 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:03:02.791460 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789894 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789898 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789903 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789908 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789913 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789917 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789921 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789925 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789930 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789934 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789938 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789942 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789946 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789951 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789955 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789959 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789964 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:03:02.791951 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.789968 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:03:02.792981 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.792964 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:03:02.792981 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.792982 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.792987 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.792992 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.792996 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793001 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793005 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793009 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793014 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793019 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793024 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793028 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793032 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793036 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793040 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793044 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793048 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793052 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793056 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793060 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793064 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:03:02.793120 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793068 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793072 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793076 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793080 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793084 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793088 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793093 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793097 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793101 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793105 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793109 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793113 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793117 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793121 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793125 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793130 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793134 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793138 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793142 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793146 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:03:02.794126 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793150 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793157 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793161 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793165 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793169 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793173 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793177 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793181 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793185 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793189 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793193 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793198 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793202 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793207 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793211 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793215 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793220 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793225 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793229 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:03:02.795025 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793233 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793240 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793246 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793251 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793256 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793261 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793266 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793270 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793275 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793280 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793284 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793288 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793293 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793297 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793301 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793306 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793310 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793314 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793318 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:03:02.795893 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793322 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793328 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793332 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793336 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793341 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793345 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793349 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793353 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793357 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793361 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793366 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793373 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793379 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793384 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793388 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793392 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793396 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793401 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793420 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:03:02.796469 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.793425 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794020 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794030 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794036 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794041 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794045 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794050 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794055 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794059 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794064 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794068 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794072 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794077 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794081 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794086 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794090 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794094 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794100 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794104 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:03:02.797003 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794108 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794113 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794117 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794121 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794125 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794130 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794134 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794138 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794143 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794147 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794151 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794155 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794160 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794164 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794168 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794173 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794177 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794181 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794186 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794190 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:03:02.797701 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794194 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794198 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794203 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794208 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794213 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794217 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794221 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794226 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794230 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794235 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794239 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794243 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794247 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794251 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794257 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794262 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794266 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794270 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794274 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794278 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:03:02.798241 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794282 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794286 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794290 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794295 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794299 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794303 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794307 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794311 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794315 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794320 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794324 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794328 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794332 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794336 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794341 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794346 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794351 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794356 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794361 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:03:02.798803 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794368 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794374 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794379 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794385 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794390 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794395 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794400 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794404 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794428 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794433 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794437 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794442 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794447 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794451 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794456 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794460 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794465 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794469 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794474 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:03:02.799318 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794478 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794483 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.794487 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794588 2552 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794599 2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794609 2552 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794617 2552 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794623 2552 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794628 2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794635 2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794641 2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794647 2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794652 2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794658 2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794663 2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794669 2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794673 2552 flags.go:64] FLAG: --cgroup-root="" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794678 2552 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794683 2552 flags.go:64] FLAG: --client-ca-file="" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794687 2552 flags.go:64] FLAG: --cloud-provider="external" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794692 2552 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794699 2552 flags.go:64] FLAG: --cluster-domain="" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794703 2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794708 2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 17:03:02.799860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794713 2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794719 2552 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794725 2552 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794730 2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794735 2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794741 2552 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794746 2552 flags.go:64] FLAG: --contention-profiling="false" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794750 2552 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794755 2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794760 2552 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794764 2552 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794771 2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794777 2552 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794782 2552 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794787 2552 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794792 2552 flags.go:64] FLAG: --enable-server="true" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794797 2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794804 2552 flags.go:64] FLAG: --event-burst="100" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794809 2552 flags.go:64] FLAG: --event-qps="50" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794814 2552 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794820 2552 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794825 2552 flags.go:64] FLAG: --eviction-hard="" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794831 2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794836 2552 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794841 2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 17:03:02.800460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794846 2552 flags.go:64] FLAG: --eviction-soft="" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794851 2552 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794856 2552 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794860 2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794866 2552 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794871 2552 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794875 2552 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794880 2552 flags.go:64] FLAG: --feature-gates="" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794886 2552 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794891 2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794896 2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794901 2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794907 2552 flags.go:64] FLAG: --healthz-port="10248" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794912 2552 flags.go:64] FLAG: --help="false" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794917 2552 flags.go:64] FLAG: --hostname-override="ip-10-0-138-185.ec2.internal" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794922 2552 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794927 2552 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794932 2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794938 2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794943 2552 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794952 2552 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794959 2552 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794964 2552 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 17:03:02.801052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794969 2552 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794974 2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794979 2552 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794984 2552 flags.go:64] FLAG: --kube-reserved="" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794988 2552 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794994 2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.794999 2552 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795004 2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795008 2552 flags.go:64] FLAG: --lock-file="" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795013 2552 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795018 2552 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795022 2552 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795031 2552 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795036 2552 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795040 2552 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795045 2552 flags.go:64] FLAG: --logging-format="text" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795049 2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795055 2552 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795060 2552 flags.go:64] FLAG: --manifest-url="" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795065 2552 flags.go:64] FLAG: --manifest-url-header="" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795072 2552 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795076 2552 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795083 2552 flags.go:64] FLAG: --max-pods="110" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795088 2552 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795093 2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 17:03:02.801732 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795098 2552 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795102 2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795107 2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795112 2552 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795119 2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795131 2552 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795136 2552 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795141 2552 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795146 2552 flags.go:64] FLAG: --pod-cidr="" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795151 2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795160 2552 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795164 2552 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795169 2552 flags.go:64] FLAG: --pods-per-core="0" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795174 2552 flags.go:64] FLAG: --port="10250" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795179 2552 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795184 2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09bee0a4b1e217061" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795189 2552 flags.go:64] FLAG: --qos-reserved="" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795194 2552 flags.go:64] FLAG: --read-only-port="10255" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795199 2552 flags.go:64] FLAG: --register-node="true" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795204 2552 flags.go:64] FLAG: --register-with-taints="" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795216 2552 flags.go:64] FLAG: --registry-burst="10" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795221 2552 flags.go:64] FLAG: --registry-qps="5" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795225 2552 flags.go:64] FLAG: --reserved-cpus="" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795230 2552 flags.go:64] FLAG: --reserved-memory="" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795236 2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 17:03:02.802382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795241 2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795247 2552 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795252 2552 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795256 2552 flags.go:64] FLAG: --runonce="false" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795261 2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795266 2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795271 2552 flags.go:64] FLAG: --seccomp-default="false" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795276 2552 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795280 2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795285 2552 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795290 2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795297 2552 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795302 2552 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795308 2552 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795313 2552 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795318 2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795323 2552 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795328 2552 flags.go:64] FLAG: --system-cgroups="" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795332 2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795341 2552 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795346 2552 flags.go:64] FLAG: --tls-cert-file="" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795351 2552 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795358 2552 flags.go:64] FLAG: --tls-min-version="" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795362 2552 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795367 2552 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 17:03:02.803008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795372 2552 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795377 2552 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795381 2552 flags.go:64] FLAG: --v="2" Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795388 2552 flags.go:64] FLAG: --version="false" Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795394 2552 flags.go:64] FLAG: --vmodule="" Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795400 2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.795422 2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795600 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795607 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795612 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795617 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795622 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795626 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795631 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795636 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795640 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795645 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795652 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795660 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795665 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795671 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:03:02.803638 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795676 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795680 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795685 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795690 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795694 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795699 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795703 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795708 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795714 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795718 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795722 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795726 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795731 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795736 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795740 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795744 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795748 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795753 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795757 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795761 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:03:02.804148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795766 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795769 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795774 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795778 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795782 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795786 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795791 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795795 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795799 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795804 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795809 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795814 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795818 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795823 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795827 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795831 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795835 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795840 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795844 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795848 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:03:02.804707 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795852 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795857 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795861 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795866 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795870 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795874 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795879 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795883 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795887 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795891 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795895 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795899 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795903 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795907 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795911 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795915 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795919 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795923 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795927 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795932 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:03:02.805194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795936 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795944 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795951 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795958 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795963 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795967 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795971 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795976 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795981 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795985 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795989 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795993 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.795998 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796002 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796007 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796011 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796015 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796020 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796024 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:03:02.805708 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796031 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796036 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796040 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796045 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796049 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.796053 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.796660 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.802980 2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.802995 2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.803015 2552 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.803021 2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803080 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803085 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803088 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:03:02.806178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803091 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803094 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803097 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803099 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803102 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803105 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803108 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803110 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803113 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803116 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803119 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803122 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803125 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803128 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803131 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803133 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803136 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803138 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803141 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:03:02.806552 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803143 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803146 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803148 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803151 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803154 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803158 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803162 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803165 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803169 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803172 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803175 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803179 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803182 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803185 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803189 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803192 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803195 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803198 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803200 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:03:02.807023 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803203 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803206 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803209 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803211 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803214 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803217 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803220 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803222 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803228 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803231 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803233 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803236 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803240 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803242 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803245 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803247 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803250 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803252 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803255 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803257 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:03:02.807495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803261 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803263 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803267 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803271 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803273 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803276 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803278 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803281 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803283 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803286 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803288 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803291 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803293 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803296 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803298 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803301 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803303 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803306 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803309 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803311 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:03:02.808015 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803314 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803317 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803320 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803322 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803325 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803327 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803329 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803332 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803335 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803337 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803340 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803343 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803345 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803348 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803350 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803353 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803355 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:03:02.808548 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803358 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.803362 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803493 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803499 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803502 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803505 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803508 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803510 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803513 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803516 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803518 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803521 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803524 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803528 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803531 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:03:02.808962 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803535 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803537 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803540 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803543 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803546 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803548 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803551 2552 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803553 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803556 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803559 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803561 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803564 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803567 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803569 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803572 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803574 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803577 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803579 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803583 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:03:02.809386 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803586 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803590 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803593 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803596 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803598 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803600 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803603 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803605 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803608 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803610 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803613 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803616 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803618 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803621 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803624 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803626 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803629 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803632 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803634 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803637 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:03:02.809868 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803639 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803642 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803645 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803647 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803650 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803653 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803655 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803658 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803660 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803663 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803665 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803668 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803670 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803673 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803675 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803678 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803680 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803683 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803685 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803688 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:03:02.810642 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803690 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803693 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803696 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803700 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803703 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803706 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803708 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803711 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803713 2552 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803716 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803719 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803722 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803724 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803726 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803729 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803732 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803734 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803736 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803739 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:03:02.811178 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803741 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803744 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803746 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803748 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803751 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803754 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803756 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:02.803759 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.803763 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.803856 2552 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.805750 2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.806848 2552 server.go:1019] "Starting client certificate rotation" Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.806943 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:03:02.811710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.806979 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:03:02.812051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.810961 2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:03:02.812365 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.812348 2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:03:02.818665 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.818649 2552 log.go:25] "Validated CRI v1 runtime API" Jul 24 17:03:02.820649 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.820635 2552 log.go:25] "Validated CRI v1 image API" Jul 24 17:03:02.822129 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.822097 2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 17:03:02.825033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.825012 2552 fs.go:135] Filesystem UUIDs: map[1a6db8b7-762a-4a9b-8b8a-1490ee38560a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d05ebf28-1214-4303-9520-aaf7064de9b9:/dev/nvme0n1p4] Jul 24 17:03:02.825112 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.825032 2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 17:03:02.831573 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.831469 2552 manager.go:216] Machine: {Timestamp:2026-07-24 17:03:02.829715343 +0000 UTC m=+0.328117880 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3152416 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f8559017bc018b9decb0f0b81841 SystemUUID:ec24f855-9017-bc01-8b9d-ecb0f0b81841 BootID:74a24df0-b787-446b-9a83-173b7811ab89 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:28:96:7a:60:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:28:96:7a:60:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:95:5b:c6:09:ec Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 17:03:02.831573 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.831567 2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 17:03:02.831708 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.831639 2552 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 17:03:02.834596 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.834573 2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 17:03:02.834757 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.834601 2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-185.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 17:03:02.835783 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.835770 2552 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 17:03:02.835843 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.835788 2552 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 17:03:02.835843 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.835799 2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:03:02.836454 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.836441 2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:03:02.837215 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.837204 2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 17:03:02.839003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.838989 2552 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:03:02.839137 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839126 2552 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 17:03:02.839202 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839163 2552 kubelet.go:486] "Attempting to sync node with API server" Jul 24 17:03:02.839202 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839178 2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 17:03:02.839288 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839202 2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 17:03:02.839288 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839216 2552 kubelet.go:398] "Adding apiserver pod source" Jul 24 17:03:02.839288 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839229 2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 17:03:02.839604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839590 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:03:02.839684 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.839612 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:03:02.840349 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.840330 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:03:02.841811 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.841789 2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 17:03:02.843807 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.843794 2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 17:03:02.843848 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.843824 2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 17:03:02.845525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845513 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 17:03:02.845525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845528 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845534 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845540 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845548 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845554 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845562 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845567 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845574 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845580 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845588 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 17:03:02.845619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.845593 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 17:03:02.846261 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.846252 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 17:03:02.846261 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.846260 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 17:03:02.850112 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.850076 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 17:03:02.850218 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.850160 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 17:03:02.850275 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.850245 2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-185.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 17:03:02.850672 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.850658 2552 server.go:1268] "Started kubelet" Jul 24 17:03:02.850784 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.850737 2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 17:03:02.850847 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.850793 2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 17:03:02.850894 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.850869 2552 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 17:03:02.851495 ip-10-0-138-185 systemd[1]: Started Kubernetes Kubelet. Jul 24 17:03:02.852703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.852686 2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 17:03:02.853434 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.853421 2552 server.go:310] "Adding debug handlers to kubelet server" Jul 24 17:03:02.857796 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.856985 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-185.ec2.internal.18c5488987efaa72 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-185.ec2.internal,UID:ip-10-0-138-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-185.ec2.internal,},FirstTimestamp:2026-07-24 17:03:02.850620018 +0000 UTC m=+0.349022554,LastTimestamp:2026-07-24 17:03:02.850620018 +0000 UTC m=+0.349022554,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-185.ec2.internal,}" Jul 24 17:03:02.858616 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.858598 2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 17:03:02.860581 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.860569 2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 17:03:02.860645 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.860630 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 17:03:02.860706 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.860688 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:02.861152 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861130 2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 17:03:02.861152 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861142 2552 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 17:03:02.861264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861157 2552 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 17:03:02.861264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861241 2552 reconstruct.go:97] "Volume reconstruction finished" Jul 24 17:03:02.861264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861252 2552 reconciler.go:29] "Reconciler: start to sync state" Jul 24 17:03:02.861391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861380 2552 factory.go:55] Registering systemd factory Jul 24 17:03:02.861391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861391 2552 factory.go:223] Registration of the systemd container factory successfully Jul 24 17:03:02.861712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861698 2552 factory.go:153] Registering CRI-O factory Jul 24 17:03:02.861712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861712 2552 factory.go:223] Registration of the crio container factory successfully Jul 24 17:03:02.861824 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861748 2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 17:03:02.861824 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861767 2552 factory.go:103] Registering Raw factory Jul 24 17:03:02.861824 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.861778 2552 manager.go:1196] Started watching for new ooms in manager Jul 24 17:03:02.862118 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.862100 2552 manager.go:349] Starting recovery of all containers Jul 24 17:03:02.863403 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.863369 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 17:03:02.863531 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.863507 2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-185.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 17:03:02.872097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.872078 2552 manager.go:354] Recovery completed Jul 24 17:03:02.878759 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.878743 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:02.881241 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.881221 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:02.881339 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.881259 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:02.881339 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.881275 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:02.881932 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.881911 2552 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 17:03:02.881932 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.881931 2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 17:03:02.882082 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.881939 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:02.882082 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.881951 2552 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:03:02.882855 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.882835 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:02.882927 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.882867 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:02.882927 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.882877 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:02.883383 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.883309 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-185.ec2.internal.18c5488989c2eab7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-185.ec2.internal,UID:ip-10-0-138-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-185.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-185.ec2.internal,},FirstTimestamp:2026-07-24 17:03:02.881241783 +0000 UTC m=+0.379644324,LastTimestamp:2026-07-24 17:03:02.881241783 +0000 UTC m=+0.379644324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-185.ec2.internal,}" Jul 24 17:03:02.883588 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.883577 2552 policy_none.go:49] "None policy: Start" Jul 24 17:03:02.883626 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.883592 2552 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 17:03:02.883626 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.883603 2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 17:03:02.884503 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.884491 2552 policy_none.go:47] "Start" Jul 24 17:03:02.887582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.887562 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kv8g" Jul 24 17:03:02.893506 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.893433 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-185.ec2.internal.18c5488989c34d22 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-185.ec2.internal,UID:ip-10-0-138-185.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-185.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-185.ec2.internal,},FirstTimestamp:2026-07-24 17:03:02.881266978 +0000 UTC m=+0.379669516,LastTimestamp:2026-07-24 17:03:02.881266978 +0000 UTC m=+0.379669516,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-185.ec2.internal,}" Jul 24 17:03:02.895371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.895353 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9kv8g" Jul 24 17:03:02.913354 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.913183 2552 manager.go:337] "Starting Device Plugin manager" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.913381 2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.913395 2552 server.go:87] "Starting device plugin registration server" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.913643 2552 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.913655 2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.913912 2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.914076 2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.914085 2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.914345 2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 17:03:02.928830 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.914400 2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:02.956360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.956327 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 17:03:02.957503 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.957486 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 17:03:02.957604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.957521 2552 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 17:03:02.957604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.957541 2552 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 17:03:02.957604 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:02.957574 2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 17:03:02.959592 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:02.959576 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:03:03.014571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.014548 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:03.015436 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.015422 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:03.015522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.015450 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:03.015522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.015460 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:03.015522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.015486 2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.024311 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.024298 2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.024359 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.024320 2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-185.ec2.internal\": node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.048139 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.048094 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.058154 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.058131 2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal"] Jul 24 17:03:03.058225 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.058218 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:03.059584 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.059570 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:03.059657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.059596 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:03.059657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.059605 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:03.060853 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.060842 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:03.060963 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.060952 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.060998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.060985 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:03.061450 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061432 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/455fb4efaf3fe9fde399a53c8ec0ea5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal\" (UID: \"455fb4efaf3fe9fde399a53c8ec0ea5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.061559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061461 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/455fb4efaf3fe9fde399a53c8ec0ea5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal\" (UID: \"455fb4efaf3fe9fde399a53c8ec0ea5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.061559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061485 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8ea5a8b31f868b5de8cabbf89b4c2b24-config\") pod \"kube-apiserver-proxy-ip-10-0-138-185.ec2.internal\" (UID: \"8ea5a8b31f868b5de8cabbf89b4c2b24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.061559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061490 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:03.061559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061518 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:03.061559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061519 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:03.061559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061533 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:03.061559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061554 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:03.061866 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.061568 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:03.063043 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.063027 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.063129 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.063051 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:03.063673 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.063656 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:03.063775 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.063679 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:03.063775 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.063689 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:03.087002 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.086984 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-185.ec2.internal\" not found" node="ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.091362 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.091346 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-185.ec2.internal\" not found" node="ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.149137 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.149109 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.162419 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.162390 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/455fb4efaf3fe9fde399a53c8ec0ea5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal\" (UID: \"455fb4efaf3fe9fde399a53c8ec0ea5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.162504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.162429 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/455fb4efaf3fe9fde399a53c8ec0ea5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal\" (UID: \"455fb4efaf3fe9fde399a53c8ec0ea5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.162504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.162445 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8ea5a8b31f868b5de8cabbf89b4c2b24-config\") pod \"kube-apiserver-proxy-ip-10-0-138-185.ec2.internal\" (UID: \"8ea5a8b31f868b5de8cabbf89b4c2b24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.162504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.162495 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/455fb4efaf3fe9fde399a53c8ec0ea5d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal\" (UID: \"455fb4efaf3fe9fde399a53c8ec0ea5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.162627 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.162554 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8ea5a8b31f868b5de8cabbf89b4c2b24-config\") pod \"kube-apiserver-proxy-ip-10-0-138-185.ec2.internal\" (UID: \"8ea5a8b31f868b5de8cabbf89b4c2b24\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.162627 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.162589 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/455fb4efaf3fe9fde399a53c8ec0ea5d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal\" (UID: \"455fb4efaf3fe9fde399a53c8ec0ea5d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.249577 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.249545 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.350298 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.350234 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.389697 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.389660 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.395295 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.395279 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" Jul 24 17:03:03.451372 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.451339 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.551779 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.551751 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.652368 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.652290 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.752802 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.752771 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.806455 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.806428 2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 17:03:03.806921 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.806586 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 17:03:03.853381 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.853349 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.899766 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.899722 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 16:58:03 +0000 UTC" deadline="2028-05-10 08:07:09.129016776 +0000 UTC" Jul 24 17:03:03.899766 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.899760 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15735h4m5.229262399s" Jul 24 17:03:03.925134 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:03.925104 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455fb4efaf3fe9fde399a53c8ec0ea5d.slice/crio-1af201857ebf27513613234863ece771e955d915aa1898825d89b697eecfdc1b WatchSource:0}: Error finding container 1af201857ebf27513613234863ece771e955d915aa1898825d89b697eecfdc1b: Status 404 returned error can't find the container with id 1af201857ebf27513613234863ece771e955d915aa1898825d89b697eecfdc1b Jul 24 17:03:03.925331 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:03.925307 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea5a8b31f868b5de8cabbf89b4c2b24.slice/crio-0cce93f609184e0215aed3b7a36f3007836485116334187b95d46991001b61bd WatchSource:0}: Error finding container 0cce93f609184e0215aed3b7a36f3007836485116334187b95d46991001b61bd: Status 404 returned error can't find the container with id 0cce93f609184e0215aed3b7a36f3007836485116334187b95d46991001b61bd Jul 24 17:03:03.929297 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.929278 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:03:03.954097 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:03.954073 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:03.960656 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.960608 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" event={"ID":"455fb4efaf3fe9fde399a53c8ec0ea5d","Type":"ContainerStarted","Data":"1af201857ebf27513613234863ece771e955d915aa1898825d89b697eecfdc1b"} Jul 24 17:03:03.961718 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.961698 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" event={"ID":"8ea5a8b31f868b5de8cabbf89b4c2b24","Type":"ContainerStarted","Data":"0cce93f609184e0215aed3b7a36f3007836485116334187b95d46991001b61bd"} Jul 24 17:03:03.985286 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:03.985264 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:03:04.059308 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.059273 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-185.ec2.internal\" not found" Jul 24 17:03:04.060466 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.060448 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:03:04.061770 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.061753 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:03:04.061841 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.061784 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:03:04.061841 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.061795 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:03:04.121582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.121551 2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:03:04.161393 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.161365 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" Jul 24 17:03:04.174957 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.174897 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:03:04.177238 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.177223 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" Jul 24 17:03:04.186222 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.186009 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:03:04.248006 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.247979 2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:03:04.830089 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.830053 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:03:04.840504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.840464 2552 apiserver.go:52] "Watching apiserver" Jul 24 17:03:04.847460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.847406 2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 17:03:04.847856 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.847825 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d","openshift-cluster-node-tuning-operator/tuned-x474w","openshift-image-registry/node-ca-fbnqh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal","openshift-multus/multus-wxwj5","openshift-dns/node-resolver-tgptf","openshift-multus/multus-additional-cni-plugins-rjwlv","openshift-multus/network-metrics-daemon-w8pxz","openshift-network-diagnostics/network-check-target-586fh","openshift-network-operator/iptables-alerter-n8lqx","openshift-ovn-kubernetes/ovnkube-node-gd69m","kube-system/global-pull-secret-syncer-zmnnx","kube-system/konnectivity-agent-lrgjl","kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal"] Jul 24 17:03:04.849444 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.849405 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.850702 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.850678 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:04.852046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.851840 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 17:03:04.852046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.851853 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qm6pz\"" Jul 24 17:03:04.852046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.851877 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.852046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.851913 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:04.852046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.851856 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 17:03:04.852756 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.852729 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 17:03:04.852850 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.852808 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 17:03:04.852914 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.852865 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kq2xt\"" Jul 24 17:03:04.853317 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.853295 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:04.853407 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.853380 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:04.854240 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.854156 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 17:03:04.854330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.854241 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 17:03:04.854522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.854502 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 17:03:04.854944 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.854505 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ckwgn\"" Jul 24 17:03:04.854944 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.854623 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.855938 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.855921 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.856575 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.856554 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5tpmz\"" Jul 24 17:03:04.856938 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.856774 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 17:03:04.857158 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.857139 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 17:03:04.857524 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.857304 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 17:03:04.857524 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.857360 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 17:03:04.857524 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.857384 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:04.857524 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.857466 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:04.857955 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.857937 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 17:03:04.858451 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.858048 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rd98\"" Jul 24 17:03:04.858451 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.858328 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 17:03:04.858792 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.858774 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.860135 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.860112 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.860744 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.860726 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 17:03:04.860879 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.860799 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 17:03:04.861340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.860972 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 17:03:04.861340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.861166 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mv8zt\"" Jul 24 17:03:04.863303 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.861934 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.865159 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.865138 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.866559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.865716 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 17:03:04.866559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.865779 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kg2bh\"" Jul 24 17:03:04.866559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.865793 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:04.866559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.866136 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 17:03:04.866559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.866180 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2494t\"" Jul 24 17:03:04.866559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.866361 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 17:03:04.867309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.867289 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:04.867409 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.867327 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 17:03:04.867409 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.867355 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:04.868342 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.868320 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 17:03:04.868601 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.868578 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 17:03:04.868967 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.868945 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 17:03:04.869311 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.869290 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 17:03:04.869385 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.869350 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 17:03:04.869443 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.869383 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 17:03:04.869570 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.869552 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bbf5x\"" Jul 24 17:03:04.870406 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870379 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-cnibin\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.870535 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870437 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-os-release\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.870608 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870549 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-socket-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.870672 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870622 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-sys-fs\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.870672 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870648 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-modprobe-d\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.870833 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870677 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tdq\" (UniqueName: \"kubernetes.io/projected/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-kube-api-access-r8tdq\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.870833 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870703 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-system-cni-dir\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.870833 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870728 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-host\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.870833 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870753 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-tuned\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.870833 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870775 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d239a832-beca-4842-b38f-fee778ed3b0d-host\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.870833 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870800 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-socket-dir-parent\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870834 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-conf-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870857 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-tmp\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870879 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5954c515-ebf7-48b7-aca4-6dbae06907a1-konnectivity-ca\") pod \"konnectivity-agent-lrgjl\" (UID: \"5954c515-ebf7-48b7-aca4-6dbae06907a1\") " pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870920 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-os-release\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.870970 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e1f622-6f4d-4174-814b-4c5945c140b5-cni-binary-copy\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871016 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-cni-multus\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871058 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-var-lib-kubelet\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.871126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871103 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-registration-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871143 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-device-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871190 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-multus-certs\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871219 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-tmp-dir\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871243 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkzw\" (UniqueName: \"kubernetes.io/projected/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-kube-api-access-4gkzw\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871269 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-systemd\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871314 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871346 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7898b308-ac60-431a-a72e-499cb0daa8f7-host-slash\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871378 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-k8s-cni-cncf-io\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871424 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-etc-kubernetes\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871470 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysconfig\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871499 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.871541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871537 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871566 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp657\" (UniqueName: \"kubernetes.io/projected/7898b308-ac60-431a-a72e-499cb0daa8f7-kube-api-access-fp657\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871590 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5954c515-ebf7-48b7-aca4-6dbae06907a1-agent-certs\") pod \"konnectivity-agent-lrgjl\" (UID: \"5954c515-ebf7-48b7-aca4-6dbae06907a1\") " pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871614 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-cni-bin\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871650 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-hostroot\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871691 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq2s7\" (UniqueName: \"kubernetes.io/projected/f4e1f622-6f4d-4174-814b-4c5945c140b5-kube-api-access-kq2s7\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871724 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871754 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qt84\" (UniqueName: \"kubernetes.io/projected/998fba82-3e20-4847-9327-483fa0f2774c-kube-api-access-6qt84\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871776 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-sys\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871799 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cnibin\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871823 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871847 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7898b308-ac60-431a-a72e-499cb0daa8f7-iptables-alerter-script\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871872 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvtcr\" (UniqueName: \"kubernetes.io/projected/d239a832-beca-4842-b38f-fee778ed3b0d-kube-api-access-lvtcr\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871895 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-kubernetes\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871933 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysctl-d\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871957 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-run\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.872142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.871980 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872002 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d239a832-beca-4842-b38f-fee778ed3b0d-serviceca\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872027 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-system-cni-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872071 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-daemon-config\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872104 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysctl-conf\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872134 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-lib-modules\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872159 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872187 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4llh\" (UniqueName: \"kubernetes.io/projected/895d8902-9011-45c2-8273-5beb236a0a5a-kube-api-access-c4llh\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872212 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-cni-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872235 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-netns\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872277 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872306 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-kubelet\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872331 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-hosts-file\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.873040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.872357 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjd8\" (UniqueName: \"kubernetes.io/projected/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-kube-api-access-qqjd8\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.880633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.880608 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:03:04.895575 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.895552 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8v7fn" Jul 24 17:03:04.906432 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.906395 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8v7fn" Jul 24 17:03:04.962231 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.962196 2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 17:03:04.972653 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972621 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7898b308-ac60-431a-a72e-499cb0daa8f7-iptables-alerter-script\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.972810 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972664 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvtcr\" (UniqueName: \"kubernetes.io/projected/d239a832-beca-4842-b38f-fee778ed3b0d-kube-api-access-lvtcr\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.972810 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972692 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-kubernetes\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.972946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972812 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysctl-d\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.972946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972851 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-run\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.972946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972876 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.972946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972907 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovn-node-metrics-cert\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.972946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972937 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d239a832-beca-4842-b38f-fee778ed3b0d-serviceca\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972962 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-system-cni-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.972990 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-daemon-config\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973016 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysctl-conf\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973043 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-lib-modules\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973067 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973093 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-dbus\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973108 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-run\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973119 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973164 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4llh\" (UniqueName: \"kubernetes.io/projected/895d8902-9011-45c2-8273-5beb236a0a5a-kube-api-access-c4llh\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973193 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-cni-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973196 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.973214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973219 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-netns\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973246 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973275 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-node-log\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973297 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7898b308-ac60-431a-a72e-499cb0daa8f7-iptables-alerter-script\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973300 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973331 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-kubelet\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973357 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-hosts-file\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973376 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-kubernetes\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973385 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjd8\" (UniqueName: \"kubernetes.io/projected/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-kube-api-access-qqjd8\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973411 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-cnibin\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973435 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-os-release\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973499 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-socket-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973510 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-netns\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973524 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysctl-d\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973537 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-sys-fs\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973597 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-var-lib-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973610 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-system-cni-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.973886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973616 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d239a832-beca-4842-b38f-fee778ed3b0d-serviceca\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973628 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovnkube-config\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973602 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-os-release\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973658 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-modprobe-d\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973695 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-kubelet\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973714 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-cni-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973770 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-hosts-file\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973784 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysctl-conf\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973785 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-lib-modules\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973855 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-modprobe-d\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973863 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-cnibin\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973872 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-etc-selinux\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.973881 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-socket-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974005 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tdq\" (UniqueName: \"kubernetes.io/projected/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-kube-api-access-r8tdq\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974059 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-sys-fs\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974096 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-system-cni-dir\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974115 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-log-socket\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.974819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974178 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-system-cni-dir\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974213 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-host\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974231 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-tuned\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974259 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-host\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974267 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbls\" (UniqueName: \"kubernetes.io/projected/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-kube-api-access-6gbls\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974308 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d239a832-beca-4842-b38f-fee778ed3b0d-host\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974337 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-socket-dir-parent\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974363 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-conf-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974388 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-tmp\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974413 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5954c515-ebf7-48b7-aca4-6dbae06907a1-konnectivity-ca\") pod \"konnectivity-agent-lrgjl\" (UID: \"5954c515-ebf7-48b7-aca4-6dbae06907a1\") " pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974421 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-daemon-config\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974407 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-socket-dir-parent\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974456 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d239a832-beca-4842-b38f-fee778ed3b0d-host\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974461 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-systemd\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974461 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-multus-conf-dir\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974502 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974537 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-os-release\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974562 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e1f622-6f4d-4174-814b-4c5945c140b5-cni-binary-copy\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.975726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974535 2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974592 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-cni-multus\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974616 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-var-lib-kubelet\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974616 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-os-release\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974636 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-registration-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974659 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-device-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974666 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-cni-multus\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974682 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-etc-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974708 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-multus-certs\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974723 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-var-lib-kubelet\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974726 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-registration-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974732 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-tmp-dir\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974769 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkzw\" (UniqueName: \"kubernetes.io/projected/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-kube-api-access-4gkzw\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974800 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-systemd\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974810 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-multus-certs\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974825 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974774 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/998fba82-3e20-4847-9327-483fa0f2774c-device-dir\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.976739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974952 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-slash\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974981 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7898b308-ac60-431a-a72e-499cb0daa8f7-host-slash\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.974991 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-tmp-dir\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975008 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-k8s-cni-cncf-io\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975016 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5954c515-ebf7-48b7-aca4-6dbae06907a1-konnectivity-ca\") pod \"konnectivity-agent-lrgjl\" (UID: \"5954c515-ebf7-48b7-aca4-6dbae06907a1\") " pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975038 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-etc-kubernetes\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975055 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-systemd\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.975077 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975084 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-etc-kubernetes\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975086 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e1f622-6f4d-4174-814b-4c5945c140b5-cni-binary-copy\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975101 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7898b308-ac60-431a-a72e-499cb0daa8f7-host-slash\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975084 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysconfig\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.975149 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs podName:895d8902-9011-45c2-8273-5beb236a0a5a nodeName:}" failed. No retries permitted until 2026-07-24 17:03:05.475129654 +0000 UTC m=+2.973532185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs") pod "network-metrics-daemon-w8pxz" (UID: "895d8902-9011-45c2-8273-5beb236a0a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975188 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-sysconfig\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975218 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-run-k8s-cni-cncf-io\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975222 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975278 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.977934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975334 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-cni-netd\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975382 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-env-overrides\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975407 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovnkube-script-lib\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975451 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp657\" (UniqueName: \"kubernetes.io/projected/7898b308-ac60-431a-a72e-499cb0daa8f7-kube-api-access-fp657\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975497 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5954c515-ebf7-48b7-aca4-6dbae06907a1-agent-certs\") pod \"konnectivity-agent-lrgjl\" (UID: \"5954c515-ebf7-48b7-aca4-6dbae06907a1\") " pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975527 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-kubelet\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975553 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-run-netns\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975577 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-cni-bin\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975603 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-hostroot\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975631 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq2s7\" (UniqueName: \"kubernetes.io/projected/f4e1f622-6f4d-4174-814b-4c5945c140b5-kube-api-access-kq2s7\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975659 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975688 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qt84\" (UniqueName: \"kubernetes.io/projected/998fba82-3e20-4847-9327-483fa0f2774c-kube-api-access-6qt84\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975713 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-cni-bin\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975760 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cni-binary-copy\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975760 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-host-var-lib-cni-bin\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975777 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-kubelet-config\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975813 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-sys\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.979067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975857 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cnibin\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975887 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975915 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-systemd-units\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975940 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-ovn\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975966 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.975859 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4e1f622-6f4d-4174-814b-4c5945c140b5-hostroot\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.976094 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-sys\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.976224 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.976234 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cnibin\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.976510 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.978700 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.978854 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-tmp\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.979406 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5954c515-ebf7-48b7-aca4-6dbae06907a1-agent-certs\") pod \"konnectivity-agent-lrgjl\" (UID: \"5954c515-ebf7-48b7-aca4-6dbae06907a1\") " pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:04.979946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.979891 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-etc-tuned\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.983203 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.983181 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:03:04.983319 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.983211 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:03:04.983319 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.983225 2552 projected.go:196] Error preparing data for projected volume kube-api-access-6xhk6 for pod openshift-network-diagnostics/network-check-target-586fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:04.983319 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:04.983293 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6 podName:37f2b5e7-f1cc-4283-b72d-e75f6d253b2b nodeName:}" failed. No retries permitted until 2026-07-24 17:03:05.483275444 +0000 UTC m=+2.981677969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhk6" (UniqueName: "kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6") pod "network-check-target-586fh" (UID: "37f2b5e7-f1cc-4283-b72d-e75f6d253b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:04.986460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.986430 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjd8\" (UniqueName: \"kubernetes.io/projected/27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8-kube-api-access-qqjd8\") pod \"multus-additional-cni-plugins-rjwlv\" (UID: \"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8\") " pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:04.987529 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.987503 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvtcr\" (UniqueName: \"kubernetes.io/projected/d239a832-beca-4842-b38f-fee778ed3b0d-kube-api-access-lvtcr\") pod \"node-ca-fbnqh\" (UID: \"d239a832-beca-4842-b38f-fee778ed3b0d\") " pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:04.987998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.987971 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tdq\" (UniqueName: \"kubernetes.io/projected/00b0fce7-06c9-45d2-bbf3-e498d0a98a6a-kube-api-access-r8tdq\") pod \"tuned-x474w\" (UID: \"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a\") " pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:04.988298 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.988279 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qt84\" (UniqueName: \"kubernetes.io/projected/998fba82-3e20-4847-9327-483fa0f2774c-kube-api-access-6qt84\") pod \"aws-ebs-csi-driver-node-wqx9d\" (UID: \"998fba82-3e20-4847-9327-483fa0f2774c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:04.989340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.989317 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq2s7\" (UniqueName: \"kubernetes.io/projected/f4e1f622-6f4d-4174-814b-4c5945c140b5-kube-api-access-kq2s7\") pod \"multus-wxwj5\" (UID: \"f4e1f622-6f4d-4174-814b-4c5945c140b5\") " pod="openshift-multus/multus-wxwj5" Jul 24 17:03:04.989916 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.989894 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4llh\" (UniqueName: \"kubernetes.io/projected/895d8902-9011-45c2-8273-5beb236a0a5a-kube-api-access-c4llh\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:04.990250 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.990231 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp657\" (UniqueName: \"kubernetes.io/projected/7898b308-ac60-431a-a72e-499cb0daa8f7-kube-api-access-fp657\") pod \"iptables-alerter-n8lqx\" (UID: \"7898b308-ac60-431a-a72e-499cb0daa8f7\") " pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:04.991672 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:04.991627 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkzw\" (UniqueName: \"kubernetes.io/projected/c68158e9-2ffc-401e-a9cd-5eb60b63d7c3-kube-api-access-4gkzw\") pod \"node-resolver-tgptf\" (UID: \"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3\") " pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:05.076606 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076574 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-log-socket\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076606 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076619 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbls\" (UniqueName: \"kubernetes.io/projected/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-kube-api-access-6gbls\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076649 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-systemd\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076674 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076698 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-etc-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076731 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-slash\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076761 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-cni-netd\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076791 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-env-overrides\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovnkube-script-lib\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076847 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-kubelet\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.076869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076867 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-run-netns\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076887 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-cni-bin\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076903 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-kubelet-config\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076925 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-systemd-units\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076947 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-ovn\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.076972 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077004 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovn-node-metrics-cert\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077039 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-dbus\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077061 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077086 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-node-log\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077101 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077139 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-var-lib-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077165 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovnkube-config\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077233 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-run-netns\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077258 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-run-ovn-kubernetes\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077287 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-node-log\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077311 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-log-socket\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.077370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077347 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-systemd-units\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077366 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-kubelet-config\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077358 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-ovn\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077393 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-cni-bin\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077291 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-var-lib-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077432 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.077464 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077497 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-systemd\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077523 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-cni-netd\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.077541 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret podName:b4985b5b-1bdb-4bd8-b250-67e63f89d01f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:05.577521371 +0000 UTC m=+3.075923898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret") pod "global-pull-secret-syncer-zmnnx" (UID: "b4985b5b-1bdb-4bd8-b250-67e63f89d01f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077464 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-slash\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077579 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-host-kubelet\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077541 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-etc-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077602 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-dbus\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077647 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-run-openvswitch\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.077994 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-env-overrides\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.078132 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovnkube-script-lib\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.078986 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.078306 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovnkube-config\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.080689 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.080629 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-ovn-node-metrics-cert\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.087919 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.087890 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbls\" (UniqueName: \"kubernetes.io/projected/f758d3bd-2ff1-41a3-b2fa-d21084c04e5e-kube-api-access-6gbls\") pod \"ovnkube-node-gd69m\" (UID: \"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e\") " pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.167023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.166984 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n8lqx" Jul 24 17:03:05.176196 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.176171 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:05.184916 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.184894 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fbnqh" Jul 24 17:03:05.193586 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.193566 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wxwj5" Jul 24 17:03:05.200133 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.200108 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tgptf" Jul 24 17:03:05.209895 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.209874 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" Jul 24 17:03:05.217591 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.217567 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-x474w" Jul 24 17:03:05.226500 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.226472 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" Jul 24 17:03:05.233330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.233311 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:05.480125 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.480035 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:05.480294 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.480182 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:05.480294 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.480255 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs podName:895d8902-9011-45c2-8273-5beb236a0a5a nodeName:}" failed. No retries permitted until 2026-07-24 17:03:06.480235157 +0000 UTC m=+3.978637702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs") pod "network-metrics-daemon-w8pxz" (UID: "895d8902-9011-45c2-8273-5beb236a0a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:05.581367 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.581321 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:05.581559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.581383 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:05.581559 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.581487 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:05.581694 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.581555 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:03:05.581694 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.581570 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret podName:b4985b5b-1bdb-4bd8-b250-67e63f89d01f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:06.581548847 +0000 UTC m=+4.079951387 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret") pod "global-pull-secret-syncer-zmnnx" (UID: "b4985b5b-1bdb-4bd8-b250-67e63f89d01f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:05.581694 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.581578 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:03:05.581694 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.581595 2552 projected.go:196] Error preparing data for projected volume kube-api-access-6xhk6 for pod openshift-network-diagnostics/network-check-target-586fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:05.581694 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.581640 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6 podName:37f2b5e7-f1cc-4283-b72d-e75f6d253b2b nodeName:}" failed. No retries permitted until 2026-07-24 17:03:06.581623998 +0000 UTC m=+4.080026528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhk6" (UniqueName: "kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6") pod "network-check-target-586fh" (UID: "37f2b5e7-f1cc-4283-b72d-e75f6d253b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:05.909418 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.909305 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:58:04 +0000 UTC" deadline="2028-05-01 09:22:02.069724216 +0000 UTC" Jul 24 17:03:05.909418 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.909357 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15520h18m56.160370096s" Jul 24 17:03:05.958813 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:05.958779 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:05.958997 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:05.958894 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:06.011607 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.011578 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e1f622_6f4d_4174_814b_4c5945c140b5.slice/crio-038c657066b69442f518f04f749c39dfe3b9a9f4defd2ff0af536fc8439882c0 WatchSource:0}: Error finding container 038c657066b69442f518f04f749c39dfe3b9a9f4defd2ff0af536fc8439882c0: Status 404 returned error can't find the container with id 038c657066b69442f518f04f749c39dfe3b9a9f4defd2ff0af536fc8439882c0 Jul 24 17:03:06.024712 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.024685 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7898b308_ac60_431a_a72e_499cb0daa8f7.slice/crio-05eec72ac61564a94108cff8b2846d3b36103b84d9cf95dae6b7f8b63e5c260c WatchSource:0}: Error finding container 05eec72ac61564a94108cff8b2846d3b36103b84d9cf95dae6b7f8b63e5c260c: Status 404 returned error can't find the container with id 05eec72ac61564a94108cff8b2846d3b36103b84d9cf95dae6b7f8b63e5c260c Jul 24 17:03:06.028314 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.028241 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d9e72d_ed1b_4f9f_bf79_30b1e641c9b8.slice/crio-689d90b7089c5e03671f7a7c1656bba4d2df084dfa9d25e3d7957ba6f215b19e WatchSource:0}: Error finding container 689d90b7089c5e03671f7a7c1656bba4d2df084dfa9d25e3d7957ba6f215b19e: Status 404 returned error can't find the container with id 689d90b7089c5e03671f7a7c1656bba4d2df084dfa9d25e3d7957ba6f215b19e Jul 24 17:03:06.030038 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.029535 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68158e9_2ffc_401e_a9cd_5eb60b63d7c3.slice/crio-6d99f28705198f90fd9a1743044f915116a3beb5f3a0bbc885233c74235dc367 WatchSource:0}: Error finding container 6d99f28705198f90fd9a1743044f915116a3beb5f3a0bbc885233c74235dc367: Status 404 returned error can't find the container with id 6d99f28705198f90fd9a1743044f915116a3beb5f3a0bbc885233c74235dc367 Jul 24 17:03:06.031582 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.031524 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf758d3bd_2ff1_41a3_b2fa_d21084c04e5e.slice/crio-73a9e5ece69b933dc4ef8472c717faafd98d27272d335225c26b63a6b31ff31d WatchSource:0}: Error finding container 73a9e5ece69b933dc4ef8472c717faafd98d27272d335225c26b63a6b31ff31d: Status 404 returned error can't find the container with id 73a9e5ece69b933dc4ef8472c717faafd98d27272d335225c26b63a6b31ff31d Jul 24 17:03:06.032402 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.032380 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd239a832_beca_4842_b38f_fee778ed3b0d.slice/crio-26260aa5fb959cf82837af64915d6c8c23dd73640d963adaed428a0ddc504211 WatchSource:0}: Error finding container 26260aa5fb959cf82837af64915d6c8c23dd73640d963adaed428a0ddc504211: Status 404 returned error can't find the container with id 26260aa5fb959cf82837af64915d6c8c23dd73640d963adaed428a0ddc504211 Jul 24 17:03:06.033238 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.033210 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b0fce7_06c9_45d2_bbf3_e498d0a98a6a.slice/crio-ea4865e491fe277cd6a0a641b4d7bc1a66fc01c3517a35ca1b6bb273419d938e WatchSource:0}: Error finding container ea4865e491fe277cd6a0a641b4d7bc1a66fc01c3517a35ca1b6bb273419d938e: Status 404 returned error can't find the container with id ea4865e491fe277cd6a0a641b4d7bc1a66fc01c3517a35ca1b6bb273419d938e Jul 24 17:03:06.034343 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.034314 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5954c515_ebf7_48b7_aca4_6dbae06907a1.slice/crio-54273c096a798b7c03bdda4182cefca468fad52480fee166505212904b252604 WatchSource:0}: Error finding container 54273c096a798b7c03bdda4182cefca468fad52480fee166505212904b252604: Status 404 returned error can't find the container with id 54273c096a798b7c03bdda4182cefca468fad52480fee166505212904b252604 Jul 24 17:03:06.035156 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:06.035067 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998fba82_3e20_4847_9327_483fa0f2774c.slice/crio-8f31e5946c0796a6f99e31ed618aaf4bdea3df888077ec4e6c1eda40933f30ca WatchSource:0}: Error finding container 8f31e5946c0796a6f99e31ed618aaf4bdea3df888077ec4e6c1eda40933f30ca: Status 404 returned error can't find the container with id 8f31e5946c0796a6f99e31ed618aaf4bdea3df888077ec4e6c1eda40933f30ca Jul 24 17:03:06.490037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.489732 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:06.490037 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.489988 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:06.490238 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.490071 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs podName:895d8902-9011-45c2-8273-5beb236a0a5a nodeName:}" failed. No retries permitted until 2026-07-24 17:03:08.490051066 +0000 UTC m=+5.988453608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs") pod "network-metrics-daemon-w8pxz" (UID: "895d8902-9011-45c2-8273-5beb236a0a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:06.590519 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.590480 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:06.590693 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.590534 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:06.590820 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.590784 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:03:06.590876 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.590834 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:03:06.590876 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.590850 2552 projected.go:196] Error preparing data for projected volume kube-api-access-6xhk6 for pod openshift-network-diagnostics/network-check-target-586fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:06.590982 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.590912 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6 podName:37f2b5e7-f1cc-4283-b72d-e75f6d253b2b nodeName:}" failed. No retries permitted until 2026-07-24 17:03:08.590893763 +0000 UTC m=+6.089296304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhk6" (UniqueName: "kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6") pod "network-check-target-586fh" (UID: "37f2b5e7-f1cc-4283-b72d-e75f6d253b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:06.591320 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.591302 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:06.591392 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.591356 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret podName:b4985b5b-1bdb-4bd8-b250-67e63f89d01f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:08.591342044 +0000 UTC m=+6.089744572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret") pod "global-pull-secret-syncer-zmnnx" (UID: "b4985b5b-1bdb-4bd8-b250-67e63f89d01f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:06.910003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.909909 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 16:58:04 +0000 UTC" deadline="2028-05-01 01:26:17.242730682 +0000 UTC" Jul 24 17:03:06.910003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.909953 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15512h23m10.332782292s" Jul 24 17:03:06.960006 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.959534 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:06.960006 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.959653 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:06.960488 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.960327 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:06.960488 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:06.960449 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:06.977450 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.977360 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fbnqh" event={"ID":"d239a832-beca-4842-b38f-fee778ed3b0d","Type":"ContainerStarted","Data":"26260aa5fb959cf82837af64915d6c8c23dd73640d963adaed428a0ddc504211"} Jul 24 17:03:06.979624 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.979598 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgptf" event={"ID":"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3","Type":"ContainerStarted","Data":"6d99f28705198f90fd9a1743044f915116a3beb5f3a0bbc885233c74235dc367"} Jul 24 17:03:06.996157 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:06.996129 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" event={"ID":"8ea5a8b31f868b5de8cabbf89b4c2b24","Type":"ContainerStarted","Data":"bd1b13a038999203161b105182f4ac7f0dcf86c6aa57b32b95abde8c57be1867"} Jul 24 17:03:07.000702 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.000676 2552 generic.go:356] "Generic (PLEG): container finished" podID="455fb4efaf3fe9fde399a53c8ec0ea5d" containerID="0f0cbdac8a84d4a6c70fa97ad90da32c1a116a102f058dc22ebe2f68ace7b1c9" exitCode=0 Jul 24 17:03:07.000797 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.000752 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" event={"ID":"455fb4efaf3fe9fde399a53c8ec0ea5d","Type":"ContainerDied","Data":"0f0cbdac8a84d4a6c70fa97ad90da32c1a116a102f058dc22ebe2f68ace7b1c9"} Jul 24 17:03:07.027244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.027213 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"73a9e5ece69b933dc4ef8472c717faafd98d27272d335225c26b63a6b31ff31d"} Jul 24 17:03:07.027989 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.027832 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-185.ec2.internal" podStartSLOduration=3.027814995 podStartE2EDuration="3.027814995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:07.01178132 +0000 UTC m=+4.510183868" watchObservedRunningTime="2026-07-24 17:03:07.027814995 +0000 UTC m=+4.526217542" Jul 24 17:03:07.032594 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.032528 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerStarted","Data":"689d90b7089c5e03671f7a7c1656bba4d2df084dfa9d25e3d7957ba6f215b19e"} Jul 24 17:03:07.038043 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.038016 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n8lqx" event={"ID":"7898b308-ac60-431a-a72e-499cb0daa8f7","Type":"ContainerStarted","Data":"05eec72ac61564a94108cff8b2846d3b36103b84d9cf95dae6b7f8b63e5c260c"} Jul 24 17:03:07.055289 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.055253 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wxwj5" event={"ID":"f4e1f622-6f4d-4174-814b-4c5945c140b5","Type":"ContainerStarted","Data":"038c657066b69442f518f04f749c39dfe3b9a9f4defd2ff0af536fc8439882c0"} Jul 24 17:03:07.057793 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.057766 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" event={"ID":"998fba82-3e20-4847-9327-483fa0f2774c","Type":"ContainerStarted","Data":"8f31e5946c0796a6f99e31ed618aaf4bdea3df888077ec4e6c1eda40933f30ca"} Jul 24 17:03:07.061596 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.061566 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x474w" event={"ID":"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a","Type":"ContainerStarted","Data":"ea4865e491fe277cd6a0a641b4d7bc1a66fc01c3517a35ca1b6bb273419d938e"} Jul 24 17:03:07.064023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.064001 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lrgjl" event={"ID":"5954c515-ebf7-48b7-aca4-6dbae06907a1","Type":"ContainerStarted","Data":"54273c096a798b7c03bdda4182cefca468fad52480fee166505212904b252604"} Jul 24 17:03:07.958063 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:07.958032 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:07.958534 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:07.958170 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:08.101360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:08.101300 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" event={"ID":"455fb4efaf3fe9fde399a53c8ec0ea5d","Type":"ContainerStarted","Data":"bc5c90b7bb94bf46986968243b6059cd1d20e97f62194679c4fe18c5da3ad2a1"} Jul 24 17:03:08.116660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:08.116559 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-185.ec2.internal" podStartSLOduration=4.11653995 podStartE2EDuration="4.11653995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:08.115997376 +0000 UTC m=+5.614399925" watchObservedRunningTime="2026-07-24 17:03:08.11653995 +0000 UTC m=+5.614942499" Jul 24 17:03:08.507310 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:08.507223 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:08.507499 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.507436 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:08.507569 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.507503 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs podName:895d8902-9011-45c2-8273-5beb236a0a5a nodeName:}" failed. No retries permitted until 2026-07-24 17:03:12.507484787 +0000 UTC m=+10.005887325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs") pod "network-metrics-daemon-w8pxz" (UID: "895d8902-9011-45c2-8273-5beb236a0a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:08.607899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:08.607858 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:08.608068 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:08.607967 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:08.608126 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.608086 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:08.608178 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.608145 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret podName:b4985b5b-1bdb-4bd8-b250-67e63f89d01f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:12.608127999 +0000 UTC m=+10.106530526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret") pod "global-pull-secret-syncer-zmnnx" (UID: "b4985b5b-1bdb-4bd8-b250-67e63f89d01f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:08.608582 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.608562 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:03:08.608674 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.608588 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:03:08.608674 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.608602 2552 projected.go:196] Error preparing data for projected volume kube-api-access-6xhk6 for pod openshift-network-diagnostics/network-check-target-586fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:08.608674 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.608648 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6 podName:37f2b5e7-f1cc-4283-b72d-e75f6d253b2b nodeName:}" failed. No retries permitted until 2026-07-24 17:03:12.608633378 +0000 UTC m=+10.107035918 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhk6" (UniqueName: "kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6") pod "network-check-target-586fh" (UID: "37f2b5e7-f1cc-4283-b72d-e75f6d253b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:08.959144 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:08.958558 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:08.959144 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.958688 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:08.959144 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:08.958975 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:08.959144 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:08.959049 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:09.958382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:09.957984 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:09.958382 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:09.958108 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:10.958557 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:10.958488 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:10.958557 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:10.958506 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:10.959098 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:10.958650 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:10.959098 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:10.959064 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:11.958289 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:11.957761 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:11.958289 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:11.957895 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:12.542086 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:12.541828 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:12.542086 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.542020 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:12.542086 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.542094 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs podName:895d8902-9011-45c2-8273-5beb236a0a5a nodeName:}" failed. No retries permitted until 2026-07-24 17:03:20.542073449 +0000 UTC m=+18.040475974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs") pod "network-metrics-daemon-w8pxz" (UID: "895d8902-9011-45c2-8273-5beb236a0a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:12.642598 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:12.642561 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:12.642762 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:12.642612 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:12.642844 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.642759 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:12.642844 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.642832 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret podName:b4985b5b-1bdb-4bd8-b250-67e63f89d01f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:20.642810928 +0000 UTC m=+18.141213457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret") pod "global-pull-secret-syncer-zmnnx" (UID: "b4985b5b-1bdb-4bd8-b250-67e63f89d01f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:12.643079 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.643058 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:03:12.643162 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.643085 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:03:12.643162 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.643099 2552 projected.go:196] Error preparing data for projected volume kube-api-access-6xhk6 for pod openshift-network-diagnostics/network-check-target-586fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:12.643162 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.643152 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6 podName:37f2b5e7-f1cc-4283-b72d-e75f6d253b2b nodeName:}" failed. No retries permitted until 2026-07-24 17:03:20.643136231 +0000 UTC m=+18.141538758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhk6" (UniqueName: "kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6") pod "network-check-target-586fh" (UID: "37f2b5e7-f1cc-4283-b72d-e75f6d253b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:12.959502 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:12.958712 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:12.959502 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.958829 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:12.959502 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:12.959286 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:12.959502 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:12.959398 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:13.957743 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:13.957708 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:13.958201 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:13.957839 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:14.958825 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:14.958785 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:14.959244 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:14.958914 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:14.959244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:14.959011 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:14.959244 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:14.959126 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:15.958299 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:15.958266 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:15.958508 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:15.958396 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:16.958826 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:16.958785 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:16.959275 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:16.958930 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:16.959275 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:16.958991 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:16.959275 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:16.959116 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:17.958654 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:17.958619 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:17.958818 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:17.958753 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:18.958505 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:18.958471 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:18.958979 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:18.958470 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:18.958979 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:18.958609 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:18.958979 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:18.958707 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:19.958700 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:19.958663 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:19.959178 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:19.958813 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:20.604363 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:20.604332 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:20.604554 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.604465 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:20.604554 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.604534 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs podName:895d8902-9011-45c2-8273-5beb236a0a5a nodeName:}" failed. No retries permitted until 2026-07-24 17:03:36.604513285 +0000 UTC m=+34.102915814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs") pod "network-metrics-daemon-w8pxz" (UID: "895d8902-9011-45c2-8273-5beb236a0a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:20.705513 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:20.705465 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:20.705513 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:20.705515 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:20.705711 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.705648 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:20.705711 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.705670 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:03:20.705711 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.705692 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:03:20.705711 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.705707 2552 projected.go:196] Error preparing data for projected volume kube-api-access-6xhk6 for pod openshift-network-diagnostics/network-check-target-586fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:20.705890 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.705718 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret podName:b4985b5b-1bdb-4bd8-b250-67e63f89d01f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:36.705704196 +0000 UTC m=+34.204106723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret") pod "global-pull-secret-syncer-zmnnx" (UID: "b4985b5b-1bdb-4bd8-b250-67e63f89d01f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:20.705890 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.705753 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6 podName:37f2b5e7-f1cc-4283-b72d-e75f6d253b2b nodeName:}" failed. No retries permitted until 2026-07-24 17:03:36.705735508 +0000 UTC m=+34.204138052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhk6" (UniqueName: "kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6") pod "network-check-target-586fh" (UID: "37f2b5e7-f1cc-4283-b72d-e75f6d253b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:20.957846 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:20.957744 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:20.958007 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.957867 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:20.958007 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:20.957916 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:20.958117 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:20.958021 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:21.958424 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:21.958384 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:21.958820 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:21.958510 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:22.959009 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:22.958976 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:22.959486 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:22.959065 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:22.959486 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:22.959077 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:22.959486 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:22.959165 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:23.958688 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:23.958654 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:23.958852 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:23.958790 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:24.959343 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:24.958961 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:24.960162 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:24.959471 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:24.960162 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:24.959139 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:24.960162 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:24.959901 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:25.141128 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.141050 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lrgjl" event={"ID":"5954c515-ebf7-48b7-aca4-6dbae06907a1","Type":"ContainerStarted","Data":"acdee19c1eed9ece1d7d2d83352830f42c4c7e202fb7f772dfb9449c620367fe"} Jul 24 17:03:25.142507 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.142475 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fbnqh" event={"ID":"d239a832-beca-4842-b38f-fee778ed3b0d","Type":"ContainerStarted","Data":"c985cf8cd396e5c4c9bca35f332807a0054751936b6566a2be941f0e4bd20a1d"} Jul 24 17:03:25.143727 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.143704 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tgptf" event={"ID":"c68158e9-2ffc-401e-a9cd-5eb60b63d7c3","Type":"ContainerStarted","Data":"90c386bb43fd70d78277aa6a14c03f1269f6370380931916e5edb3b04069ed43"} Jul 24 17:03:25.146458 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.146437 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"de7bc5af3f1ac710ec01289e8d1e83533b4ee2cd7e29b9042caae470a3afeba8"} Jul 24 17:03:25.146557 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.146494 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"67db1bc3d7776d7b7fc9a46c1832b3b1afda0672dcbc4ce862aa749101be9a96"} Jul 24 17:03:25.146557 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.146536 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"cd676dd249145691e163d6f05d36e6377c627b70e1e21c8f559a17f92bd4ef05"} Jul 24 17:03:25.146680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.146579 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"e9c2ae389ea9e5fea949be6d027f0c13159656ac711a2c1a191f71e469dd6d0d"} Jul 24 17:03:25.146680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.146621 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"55d82f0728dcd08b487b4b86a0f4d42f12dc8f59a03560946a4d572d85499418"} Jul 24 17:03:25.146680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.146665 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"982dc142f8fd0a96c083c203b59cba943e5332517dd1adae119dd80eba0ac357"} Jul 24 17:03:25.147823 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.147801 2552 generic.go:356] "Generic (PLEG): container finished" podID="27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8" containerID="d2ea49f3d9fdb3abe1ef06ac3d528b2f0ebfa5d11f79e401cb9d664526de37ad" exitCode=0 Jul 24 17:03:25.147910 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.147863 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerDied","Data":"d2ea49f3d9fdb3abe1ef06ac3d528b2f0ebfa5d11f79e401cb9d664526de37ad"} Jul 24 17:03:25.151610 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.151026 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wxwj5" event={"ID":"f4e1f622-6f4d-4174-814b-4c5945c140b5","Type":"ContainerStarted","Data":"aa2f8e773f4bd92bea8f96bd001622114a9f3c6457875a851dcfe79a6dcacf6f"} Jul 24 17:03:25.152548 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.152522 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" event={"ID":"998fba82-3e20-4847-9327-483fa0f2774c","Type":"ContainerStarted","Data":"0d0fb1d84255b32132d6d702760b6d78dcb06f34754c69c27ae60d97f9833221"} Jul 24 17:03:25.153807 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.153785 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-x474w" event={"ID":"00b0fce7-06c9-45d2-bbf3-e498d0a98a6a","Type":"ContainerStarted","Data":"8da6b905d66c81aabf21cacf8c3907bb19faccc3e013b05a165de36d635cde0d"} Jul 24 17:03:25.158468 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.158431 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lrgjl" podStartSLOduration=11.751340887 podStartE2EDuration="22.158400545s" totalImagesPullingTime="10.407059658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:25.158037864 +0000 UTC m=+22.656440410" watchObservedRunningTime="2026-07-24 17:03:25.158400545 +0000 UTC m=+22.656803089" Jul 24 17:03:25.171259 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.171222 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fbnqh" podStartSLOduration=3.960683379 podStartE2EDuration="22.171207201s" totalImagesPullingTime="18.210523822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:25.170998129 +0000 UTC m=+22.669400674" watchObservedRunningTime="2026-07-24 17:03:25.171207201 +0000 UTC m=+22.669609747" Jul 24 17:03:25.211515 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.211472 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wxwj5" podStartSLOduration=3.937733926 podStartE2EDuration="22.211458726s" totalImagesPullingTime="18.2737248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:25.188122659 +0000 UTC m=+22.686525207" watchObservedRunningTime="2026-07-24 17:03:25.211458726 +0000 UTC m=+22.709861279" Jul 24 17:03:25.231422 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.231369 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tgptf" podStartSLOduration=4.017878126 podStartE2EDuration="22.231352517s" totalImagesPullingTime="18.213474391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:25.231110465 +0000 UTC m=+22.729513011" watchObservedRunningTime="2026-07-24 17:03:25.231352517 +0000 UTC m=+22.729755062" Jul 24 17:03:25.248723 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.248687 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-x474w" podStartSLOduration=4.033741211 podStartE2EDuration="22.248674561s" totalImagesPullingTime="18.21493335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:25.247982713 +0000 UTC m=+22.746385258" watchObservedRunningTime="2026-07-24 17:03:25.248674561 +0000 UTC m=+22.747077107" Jul 24 17:03:25.798313 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.798286 2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 17:03:25.928122 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.927967 2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:03:25.798309239Z","UUID":"bac7ffe8-a4f7-43c9-b14d-095996e5e786","Handler":null,"Name":"","Endpoint":""} Jul 24 17:03:25.929268 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.929248 2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 17:03:25.929373 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.929281 2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 17:03:25.958374 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:25.958350 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:25.958497 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:25.958453 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:26.157614 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:26.157527 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n8lqx" event={"ID":"7898b308-ac60-431a-a72e-499cb0daa8f7","Type":"ContainerStarted","Data":"4f3e7d2a1967ef35d6e8a02df61a74401782d83b3a3e17e847086f157de3c63b"} Jul 24 17:03:26.159483 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:26.159457 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" event={"ID":"998fba82-3e20-4847-9327-483fa0f2774c","Type":"ContainerStarted","Data":"c1529db7cb4b9e3f4dd31a6d85e67195529c6589a25a62c4a87fe4af16ad907c"} Jul 24 17:03:26.171284 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:26.171245 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n8lqx" podStartSLOduration=4.954624257 podStartE2EDuration="23.17123294s" totalImagesPullingTime="18.216608683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:26.170888166 +0000 UTC m=+23.669290711" watchObservedRunningTime="2026-07-24 17:03:26.17123294 +0000 UTC m=+23.669635485" Jul 24 17:03:26.958753 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:26.958722 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:26.959160 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:26.958839 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:26.959160 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:26.958895 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:26.959160 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:26.959018 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:27.036736 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:27.036711 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:27.037347 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:27.037327 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:27.162087 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:27.162058 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:27.163108 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:27.163087 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lrgjl" Jul 24 17:03:27.958571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:27.958390 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:27.958882 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:27.958654 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:28.166381 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:28.166334 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" event={"ID":"998fba82-3e20-4847-9327-483fa0f2774c","Type":"ContainerStarted","Data":"c636173122d47f765534826ef68f7555dadd25c6525418bd40246e23d8daab55"} Jul 24 17:03:28.170108 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:28.170083 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"acc53a7cffdb7e3f0c549cac76e77396fe8e2ad97aac1ca0f32249a0ac7d5e52"} Jul 24 17:03:28.182767 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:28.182726 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wqx9d" podStartSLOduration=4.336786909 podStartE2EDuration="25.182711114s" totalImagesPullingTime="20.845924205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:28.182164546 +0000 UTC m=+25.680567091" watchObservedRunningTime="2026-07-24 17:03:28.182711114 +0000 UTC m=+25.681113671" Jul 24 17:03:28.957920 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:28.957888 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:28.958095 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:28.958032 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:28.958162 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:28.958095 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:28.958227 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:28.958207 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:29.958670 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:29.958514 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:29.959212 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:29.958748 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:30.179584 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.179490 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" event={"ID":"f758d3bd-2ff1-41a3-b2fa-d21084c04e5e","Type":"ContainerStarted","Data":"b15d5afb3a602c0468d7bb55a162a7db791327ffc7f0fbf9336e654176e97d8d"} Jul 24 17:03:30.179799 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.179773 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:30.179931 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.179854 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:30.179931 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.179895 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:30.181344 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.181308 2552 generic.go:356] "Generic (PLEG): container finished" podID="27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8" containerID="58e6f5d58b84c1e3a4a1ae854ad04100976820f73c94f3fb165eb8a1761e124b" exitCode=0 Jul 24 17:03:30.181456 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.181359 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerDied","Data":"58e6f5d58b84c1e3a4a1ae854ad04100976820f73c94f3fb165eb8a1761e124b"} Jul 24 17:03:30.195059 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.195038 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:30.195177 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.195134 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:03:30.206338 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.206300 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" podStartSLOduration=8.934076305 podStartE2EDuration="27.206288228s" totalImagesPullingTime="18.272211923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:30.205754547 +0000 UTC m=+27.704157131" watchObservedRunningTime="2026-07-24 17:03:30.206288228 +0000 UTC m=+27.704690774" Jul 24 17:03:30.958309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.957819 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:30.958309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:30.957968 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:30.958309 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:30.958094 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:30.958309 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:30.957964 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:31.707396 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:31.707368 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8pxz"] Jul 24 17:03:31.707574 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:31.707536 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:31.707648 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:31.707629 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:31.710681 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:31.710651 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmnnx"] Jul 24 17:03:31.710798 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:31.710752 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:31.710873 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:31.710848 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:31.711477 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:31.711452 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-586fh"] Jul 24 17:03:31.711569 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:31.711552 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:31.711669 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:31.711645 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:32.187728 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:32.187692 2552 generic.go:356] "Generic (PLEG): container finished" podID="27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8" containerID="1d097bf3108d4e486735983e3e2f7d9f12c01e931cedc3203564508947b4e8af" exitCode=0 Jul 24 17:03:32.188116 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:32.187775 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerDied","Data":"1d097bf3108d4e486735983e3e2f7d9f12c01e931cedc3203564508947b4e8af"} Jul 24 17:03:32.958733 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:32.958704 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:32.958937 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:32.958817 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:32.958937 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:32.958880 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:32.959064 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:32.958992 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:33.957893 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:33.957730 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:33.958238 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:33.957965 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:34.194248 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:34.194208 2552 generic.go:356] "Generic (PLEG): container finished" podID="27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8" containerID="29d1fc3d2eb715765dddd25ca074657f31131becde1e68b6ef857581b013d066" exitCode=0 Jul 24 17:03:34.194400 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:34.194255 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerDied","Data":"29d1fc3d2eb715765dddd25ca074657f31131becde1e68b6ef857581b013d066"} Jul 24 17:03:34.958474 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:34.958438 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:34.958892 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:34.958446 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:34.958892 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:34.958591 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:34.958892 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:34.958660 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:35.958356 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:35.958319 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:35.958547 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:35.958477 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-586fh" podUID="37f2b5e7-f1cc-4283-b72d-e75f6d253b2b" Jul 24 17:03:36.628017 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:36.627980 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:36.628202 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.628146 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:36.628257 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.628220 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs podName:895d8902-9011-45c2-8273-5beb236a0a5a nodeName:}" failed. No retries permitted until 2026-07-24 17:04:08.628202464 +0000 UTC m=+66.126604988 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs") pod "network-metrics-daemon-w8pxz" (UID: "895d8902-9011-45c2-8273-5beb236a0a5a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:03:36.729118 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:36.729080 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:36.729298 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:36.729181 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:36.729298 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.729272 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:36.729298 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.729282 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:03:36.729447 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.729306 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:03:36.729447 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.729321 2552 projected.go:196] Error preparing data for projected volume kube-api-access-6xhk6 for pod openshift-network-diagnostics/network-check-target-586fh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:36.729447 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.729330 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret podName:b4985b5b-1bdb-4bd8-b250-67e63f89d01f nodeName:}" failed. No retries permitted until 2026-07-24 17:04:08.729313554 +0000 UTC m=+66.227716081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret") pod "global-pull-secret-syncer-zmnnx" (UID: "b4985b5b-1bdb-4bd8-b250-67e63f89d01f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:03:36.729447 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.729375 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6 podName:37f2b5e7-f1cc-4283-b72d-e75f6d253b2b nodeName:}" failed. No retries permitted until 2026-07-24 17:04:08.72935894 +0000 UTC m=+66.227761481 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6xhk6" (UniqueName: "kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6") pod "network-check-target-586fh" (UID: "37f2b5e7-f1cc-4283-b72d-e75f6d253b2b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:03:36.958882 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:36.958837 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:36.959351 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:36.958837 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:36.959351 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.958974 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmnnx" podUID="b4985b5b-1bdb-4bd8-b250-67e63f89d01f" Jul 24 17:03:36.959351 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:36.959031 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8pxz" podUID="895d8902-9011-45c2-8273-5beb236a0a5a" Jul 24 17:03:37.805496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.805454 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-185.ec2.internal" event="NodeReady" Jul 24 17:03:37.805670 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.805618 2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 17:03:37.850477 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.850444 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6566bdd846-59cjr"] Jul 24 17:03:37.855454 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.855432 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.858018 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.857990 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 17:03:37.858247 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.858231 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v9fwg\"" Jul 24 17:03:37.858467 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.858448 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 17:03:37.858860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.858838 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d8cbf98d5-bg4mh"] Jul 24 17:03:37.859047 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.859028 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 17:03:37.862367 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.862348 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm"] Jul 24 17:03:37.862498 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.862479 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:37.863983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.863961 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 17:03:37.864733 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.864641 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 17:03:37.864733 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.864656 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 17:03:37.864900 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.864741 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 17:03:37.864900 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.864884 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 17:03:37.865015 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.864996 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.865116 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.865104 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.865268 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.865244 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9tk6v\"" Jul 24 17:03:37.867731 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.867715 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg"] Jul 24 17:03:37.867920 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.867902 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" Jul 24 17:03:37.870037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.869987 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6kzb8\"" Jul 24 17:03:37.870037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.870055 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.870037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.870221 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.870793 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.870770 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk"] Jul 24 17:03:37.870933 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.870915 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:37.873667 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.872811 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9pb4g\"" Jul 24 17:03:37.873667 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.873022 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.873667 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.873308 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.874155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.874026 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 17:03:37.874155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.874048 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 17:03:37.878764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.878470 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj"] Jul 24 17:03:37.878764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.878688 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:37.881595 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.881508 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 17:03:37.881714 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.881566 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-8rxmc\"" Jul 24 17:03:37.881912 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.881843 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.881912 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.881904 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.884510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.884492 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sq9g2"] Jul 24 17:03:37.884670 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.884642 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:37.886541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.886525 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 17:03:37.886781 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.886763 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.886869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.886782 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.887037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.887015 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 17:03:37.887259 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.887239 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xhhg6\"" Jul 24 17:03:37.888845 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.888589 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6566bdd846-59cjr"] Jul 24 17:03:37.888845 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.888635 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj"] Jul 24 17:03:37.888845 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.888788 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk"] Jul 24 17:03:37.889209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.888878 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg"] Jul 24 17:03:37.889209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.888923 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d8cbf98d5-bg4mh"] Jul 24 17:03:37.889209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.888943 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm"] Jul 24 17:03:37.889209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.889113 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:37.890668 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.890644 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sq9g2"] Jul 24 17:03:37.891549 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.891530 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 17:03:37.891837 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.891821 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gnbfc\"" Jul 24 17:03:37.892055 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.892035 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 17:03:37.916244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.916223 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz"] Jul 24 17:03:37.919821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.919799 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:37.920542 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.920525 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-455nt"] Jul 24 17:03:37.923149 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.923128 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 17:03:37.923615 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.923589 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.923849 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.923696 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:37.924498 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.924481 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mkptd"] Jul 24 17:03:37.925602 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.925565 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.925602 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.925601 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6d47d\"" Jul 24 17:03:37.926102 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.926081 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 17:03:37.926340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.926326 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xzjts\"" Jul 24 17:03:37.926821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.926804 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 17:03:37.926908 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.926843 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 17:03:37.927128 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.927111 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.927571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.927545 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c5bfp"] Jul 24 17:03:37.927824 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.927735 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:37.928349 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.928330 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.930588 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.930557 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ln2rz\"" Jul 24 17:03:37.930704 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.930589 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.930704 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.930665 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 17:03:37.930838 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.930753 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:37.933117 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.933098 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 17:03:37.934798 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.934777 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz"] Jul 24 17:03:37.935155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.935134 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 17:03:37.935509 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.935490 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.936392 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.936322 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 17:03:37.937170 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.937146 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.937719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.937697 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-455nt"] Jul 24 17:03:37.938116 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.938048 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.938203 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.938154 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-trusted-ca\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.938393 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.938287 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-image-registry-private-configuration\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.938699 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.938566 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d46f7\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-kube-api-access-d46f7\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.938991 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.938677 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-certificates\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.939236 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.938929 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-bound-sa-token\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.939503 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.939399 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4e20cf2-9271-4dfd-9143-0101267953ab-ca-trust-extracted\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.939632 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.939538 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.939741 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.939637 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x7c4f\"" Jul 24 17:03:37.940164 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.939899 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-installation-pull-secrets\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:37.946826 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.946800 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 17:03:37.950046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.950022 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mkptd"] Jul 24 17:03:37.950430 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.950397 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c5bfp"] Jul 24 17:03:37.957775 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.957757 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:03:37.961131 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.961113 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 17:03:37.961586 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.961129 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 17:03:37.961586 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:37.961244 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lpvrb\"" Jul 24 17:03:38.001871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.001800 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284"] Jul 24 17:03:38.005315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.005291 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" Jul 24 17:03:38.007401 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.007380 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gkd79\"" Jul 24 17:03:38.017836 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.017812 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284"] Jul 24 17:03:38.040430 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040386 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d20e785-46d3-4a00-a8a6-50a30445dbc8-cert\") pod \"ingress-canary-mkptd\" (UID: \"5d20e785-46d3-4a00-a8a6-50a30445dbc8\") " pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:38.040556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040445 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-trusted-ca\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.040556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040478 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmn6\" (UniqueName: \"kubernetes.io/projected/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-kube-api-access-2cmn6\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.040556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040505 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45bb5108-6076-46d7-9421-ca8ea674fe31-config-volume\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.040556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040545 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d46f7\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-kube-api-access-d46f7\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.040761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040572 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d5333a0-2111-4134-b27c-8a6d49e53bc0-config\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.040761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040602 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.040761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040626 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2gl\" (UniqueName: \"kubernetes.io/projected/477521c4-4d67-4c6e-95b9-44dfb810a695-kube-api-access-xl2gl\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.040761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040664 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-bound-sa-token\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.040761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040694 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-stats-auth\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.040761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040720 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfa92a22-0110-4c21-8ec9-3de926b4cba3-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040760 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45bb5108-6076-46d7-9421-ca8ea674fe31-tmp-dir\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040807 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d5333a0-2111-4134-b27c-8a6d49e53bc0-trusted-ca\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040841 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788j4\" (UniqueName: \"kubernetes.io/projected/bfa92a22-0110-4c21-8ec9-3de926b4cba3-kube-api-access-788j4\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040868 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477521c4-4d67-4c6e-95b9-44dfb810a695-serving-cert\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040891 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040940 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.040987 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74kx\" (UniqueName: \"kubernetes.io/projected/bd5a2cba-dca8-4af9-b0cf-c735185b0386-kube-api-access-j74kx\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041014 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.041038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041041 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlljv\" (UniqueName: \"kubernetes.io/projected/45bb5108-6076-46d7-9421-ca8ea674fe31-kube-api-access-rlljv\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041075 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041106 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3265b58a-6a78-4afa-ba20-533d76387071-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041154 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxrtp\" (UniqueName: \"kubernetes.io/projected/69009bfd-dd6d-4838-9981-c3b080dbadd6-kube-api-access-jxrtp\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041186 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqslp\" (UniqueName: \"kubernetes.io/projected/998d1e8e-6a69-47c4-a035-bbbdb44d68bc-kube-api-access-dqslp\") pod \"volume-data-source-validator-549c6d54cf-jf4vm\" (UID: \"998d1e8e-6a69-47c4-a035-bbbdb44d68bc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.041163 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041212 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfa92a22-0110-4c21-8ec9-3de926b4cba3-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.041219 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6566bdd846-59cjr: secret "image-registry-tls" not found Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041243 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-image-registry-private-configuration\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041270 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3265b58a-6a78-4afa-ba20-533d76387071-tmp\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.041286 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls podName:c4e20cf2-9271-4dfd-9143-0101267953ab nodeName:}" failed. No retries permitted until 2026-07-24 17:03:38.541268044 +0000 UTC m=+36.039670572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls") pod "image-registry-6566bdd846-59cjr" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab") : secret "image-registry-tls" not found Jul 24 17:03:38.041360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041333 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-certificates\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041367 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vbh\" (UniqueName: \"kubernetes.io/projected/5d20e785-46d3-4a00-a8a6-50a30445dbc8-kube-api-access-h2vbh\") pod \"ingress-canary-mkptd\" (UID: \"5d20e785-46d3-4a00-a8a6-50a30445dbc8\") " pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041426 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041457 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2b4\" (UniqueName: \"kubernetes.io/projected/3265b58a-6a78-4afa-ba20-533d76387071-kube-api-access-9c2b4\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041486 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4e20cf2-9271-4dfd-9143-0101267953ab-ca-trust-extracted\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041458 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-trusted-ca\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041510 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041586 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5333a0-2111-4134-b27c-8a6d49e53bc0-serving-cert\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041616 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-installation-pull-secrets\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041642 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvd8\" (UniqueName: \"kubernetes.io/projected/9d5333a0-2111-4134-b27c-8a6d49e53bc0-kube-api-access-6xvd8\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041664 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-default-certificate\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041686 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477521c4-4d67-4c6e-95b9-44dfb810a695-config\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041737 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3265b58a-6a78-4afa-ba20-533d76387071-snapshots\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041765 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3265b58a-6a78-4afa-ba20-533d76387071-service-ca-bundle\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041798 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3265b58a-6a78-4afa-ba20-533d76387071-serving-cert\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.041884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.041801 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4e20cf2-9271-4dfd-9143-0101267953ab-ca-trust-extracted\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.042469 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.042012 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-certificates\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.045872 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.045850 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-installation-pull-secrets\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.045997 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.045873 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-image-registry-private-configuration\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.053545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.053519 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-bound-sa-token\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.054147 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.054124 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d46f7\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-kube-api-access-d46f7\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.097940 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.097902 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:38.142345 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142302 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vbh\" (UniqueName: \"kubernetes.io/projected/5d20e785-46d3-4a00-a8a6-50a30445dbc8-kube-api-access-h2vbh\") pod \"ingress-canary-mkptd\" (UID: \"5d20e785-46d3-4a00-a8a6-50a30445dbc8\") " pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142362 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142392 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2b4\" (UniqueName: \"kubernetes.io/projected/3265b58a-6a78-4afa-ba20-533d76387071-kube-api-access-9c2b4\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142425 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142449 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5333a0-2111-4134-b27c-8a6d49e53bc0-serving-cert\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142475 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvd8\" (UniqueName: \"kubernetes.io/projected/9d5333a0-2111-4134-b27c-8a6d49e53bc0-kube-api-access-6xvd8\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142500 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-default-certificate\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142523 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477521c4-4d67-4c6e-95b9-44dfb810a695-config\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.142537 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:03:38.142563 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142552 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3265b58a-6a78-4afa-ba20-533d76387071-snapshots\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142580 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3265b58a-6a78-4afa-ba20-533d76387071-service-ca-bundle\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.142615 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls podName:bd5a2cba-dca8-4af9-b0cf-c735185b0386 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:38.642595207 +0000 UTC m=+36.140997736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls") pod "cluster-samples-operator-6748489656-j7tbk" (UID: "bd5a2cba-dca8-4af9-b0cf-c735185b0386") : secret "samples-operator-tls" not found Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142643 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3265b58a-6a78-4afa-ba20-533d76387071-serving-cert\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.142678 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.142733 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls podName:0e0f63bd-220f-46f4-b52c-60dccf2f6e2f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:38.642721174 +0000 UTC m=+36.141123699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vf6rz" (UID: "0e0f63bd-220f-46f4-b52c-60dccf2f6e2f") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142677 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d20e785-46d3-4a00-a8a6-50a30445dbc8-cert\") pod \"ingress-canary-mkptd\" (UID: \"5d20e785-46d3-4a00-a8a6-50a30445dbc8\") " pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142908 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmn6\" (UniqueName: \"kubernetes.io/projected/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-kube-api-access-2cmn6\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142944 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45bb5108-6076-46d7-9421-ca8ea674fe31-config-volume\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.142983 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d5333a0-2111-4134-b27c-8a6d49e53bc0-config\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.143022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143005 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143033 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2gl\" (UniqueName: \"kubernetes.io/projected/477521c4-4d67-4c6e-95b9-44dfb810a695-kube-api-access-xl2gl\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143073 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-stats-auth\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143099 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfa92a22-0110-4c21-8ec9-3de926b4cba3-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143127 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45bb5108-6076-46d7-9421-ca8ea674fe31-tmp-dir\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143158 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d5333a0-2111-4134-b27c-8a6d49e53bc0-trusted-ca\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143183 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788j4\" (UniqueName: \"kubernetes.io/projected/bfa92a22-0110-4c21-8ec9-3de926b4cba3-kube-api-access-788j4\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143207 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477521c4-4d67-4c6e-95b9-44dfb810a695-serving-cert\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143211 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3265b58a-6a78-4afa-ba20-533d76387071-snapshots\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143229 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143272 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143302 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74kx\" (UniqueName: \"kubernetes.io/projected/bd5a2cba-dca8-4af9-b0cf-c735185b0386-kube-api-access-j74kx\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.143311 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143332 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143355 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/477521c4-4d67-4c6e-95b9-44dfb810a695-config\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143372 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3265b58a-6a78-4afa-ba20-533d76387071-service-ca-bundle\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.143545 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.143388 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls podName:45bb5108-6076-46d7-9421-ca8ea674fe31 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:38.643370263 +0000 UTC m=+36.141772788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls") pod "dns-default-sq9g2" (UID: "45bb5108-6076-46d7-9421-ca8ea674fe31") : secret "dns-default-metrics-tls" not found Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.143590 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.143635 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:38.643620674 +0000 UTC m=+36.142023201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : secret "router-metrics-certs-default" not found Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143757 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d5333a0-2111-4134-b27c-8a6d49e53bc0-config\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144144 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.143358 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlljv\" (UniqueName: \"kubernetes.io/projected/45bb5108-6076-46d7-9421-ca8ea674fe31-kube-api-access-rlljv\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144223 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3265b58a-6a78-4afa-ba20-533d76387071-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144271 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxrtp\" (UniqueName: \"kubernetes.io/projected/69009bfd-dd6d-4838-9981-c3b080dbadd6-kube-api-access-jxrtp\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144297 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqslp\" (UniqueName: \"kubernetes.io/projected/998d1e8e-6a69-47c4-a035-bbbdb44d68bc-kube-api-access-dqslp\") pod \"volume-data-source-validator-549c6d54cf-jf4vm\" (UID: \"998d1e8e-6a69-47c4-a035-bbbdb44d68bc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144308 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/45bb5108-6076-46d7-9421-ca8ea674fe31-config-volume\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144325 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfa92a22-0110-4c21-8ec9-3de926b4cba3-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.144371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144359 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3265b58a-6a78-4afa-ba20-533d76387071-tmp\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.144943 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144391 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxxv\" (UniqueName: \"kubernetes.io/projected/5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0-kube-api-access-5sxxv\") pod \"network-check-source-7d88cb55d8-qt284\" (UID: \"5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" Jul 24 17:03:38.144943 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.144434 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:38.644394529 +0000 UTC m=+36.142797078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : configmap references non-existent config key: service-ca.crt Jul 24 17:03:38.144943 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.144787 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d5333a0-2111-4134-b27c-8a6d49e53bc0-trusted-ca\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.145104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.145053 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45bb5108-6076-46d7-9421-ca8ea674fe31-tmp-dir\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.145864 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.145267 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3265b58a-6a78-4afa-ba20-533d76387071-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.145864 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.145632 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3265b58a-6a78-4afa-ba20-533d76387071-tmp\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.145864 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.145802 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfa92a22-0110-4c21-8ec9-3de926b4cba3-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.146326 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.146286 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-stats-auth\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.146602 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.146576 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d5333a0-2111-4134-b27c-8a6d49e53bc0-serving-cert\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.146667 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.146639 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d20e785-46d3-4a00-a8a6-50a30445dbc8-cert\") pod \"ingress-canary-mkptd\" (UID: \"5d20e785-46d3-4a00-a8a6-50a30445dbc8\") " pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:38.146917 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.146898 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfa92a22-0110-4c21-8ec9-3de926b4cba3-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.147121 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.147102 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-default-certificate\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.147954 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.147932 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/477521c4-4d67-4c6e-95b9-44dfb810a695-serving-cert\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.148044 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.148028 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3265b58a-6a78-4afa-ba20-533d76387071-serving-cert\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.160808 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.160787 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vbh\" (UniqueName: \"kubernetes.io/projected/5d20e785-46d3-4a00-a8a6-50a30445dbc8-kube-api-access-h2vbh\") pod \"ingress-canary-mkptd\" (UID: \"5d20e785-46d3-4a00-a8a6-50a30445dbc8\") " pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:38.167073 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.166582 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlljv\" (UniqueName: \"kubernetes.io/projected/45bb5108-6076-46d7-9421-ca8ea674fe31-kube-api-access-rlljv\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.167073 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.166724 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqslp\" (UniqueName: \"kubernetes.io/projected/998d1e8e-6a69-47c4-a035-bbbdb44d68bc-kube-api-access-dqslp\") pod \"volume-data-source-validator-549c6d54cf-jf4vm\" (UID: \"998d1e8e-6a69-47c4-a035-bbbdb44d68bc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" Jul 24 17:03:38.167073 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.167024 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxrtp\" (UniqueName: \"kubernetes.io/projected/69009bfd-dd6d-4838-9981-c3b080dbadd6-kube-api-access-jxrtp\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.167073 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.167041 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74kx\" (UniqueName: \"kubernetes.io/projected/bd5a2cba-dca8-4af9-b0cf-c735185b0386-kube-api-access-j74kx\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:38.167559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.167540 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmn6\" (UniqueName: \"kubernetes.io/projected/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-kube-api-access-2cmn6\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.168290 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.168272 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2b4\" (UniqueName: \"kubernetes.io/projected/3265b58a-6a78-4afa-ba20-533d76387071-kube-api-access-9c2b4\") pod \"insights-operator-5885fdb759-455nt\" (UID: \"3265b58a-6a78-4afa-ba20-533d76387071\") " pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.168495 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.168475 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvd8\" (UniqueName: \"kubernetes.io/projected/9d5333a0-2111-4134-b27c-8a6d49e53bc0-kube-api-access-6xvd8\") pod \"console-operator-584dcb6b54-c5bfp\" (UID: \"9d5333a0-2111-4134-b27c-8a6d49e53bc0\") " pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.168671 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.168655 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-788j4\" (UniqueName: \"kubernetes.io/projected/bfa92a22-0110-4c21-8ec9-3de926b4cba3-kube-api-access-788j4\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg\" (UID: \"bfa92a22-0110-4c21-8ec9-3de926b4cba3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.169027 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.169000 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2gl\" (UniqueName: \"kubernetes.io/projected/477521c4-4d67-4c6e-95b9-44dfb810a695-kube-api-access-xl2gl\") pod \"service-ca-operator-5d6c75f84f-dnlgj\" (UID: \"477521c4-4d67-4c6e-95b9-44dfb810a695\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.191544 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.191519 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" Jul 24 17:03:38.197542 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.197516 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" Jul 24 17:03:38.216725 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.216688 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:38.218433 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.218397 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" Jul 24 17:03:38.242441 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.242402 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-455nt" Jul 24 17:03:38.245275 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.245147 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxxv\" (UniqueName: \"kubernetes.io/projected/5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0-kube-api-access-5sxxv\") pod \"network-check-source-7d88cb55d8-qt284\" (UID: \"5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" Jul 24 17:03:38.253286 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.253227 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mkptd" Jul 24 17:03:38.255889 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.255861 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxxv\" (UniqueName: \"kubernetes.io/projected/5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0-kube-api-access-5sxxv\") pod \"network-check-source-7d88cb55d8-qt284\" (UID: \"5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" Jul 24 17:03:38.259201 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.259180 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:38.315899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.315865 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" Jul 24 17:03:38.548832 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.548746 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:38.548971 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.548936 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:03:38.548971 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.548957 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6566bdd846-59cjr: secret "image-registry-tls" not found Jul 24 17:03:38.549058 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.549024 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls podName:c4e20cf2-9271-4dfd-9143-0101267953ab nodeName:}" failed. No retries permitted until 2026-07-24 17:03:39.549002177 +0000 UTC m=+37.047404705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls") pod "image-registry-6566bdd846-59cjr" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab") : secret "image-registry-tls" not found Jul 24 17:03:38.650211 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.650167 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.650375 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.650240 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:38.650375 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.650270 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:38.650375 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.650367 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:38.650546 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.650478 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650602 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650680 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls podName:45bb5108-6076-46d7-9421-ca8ea674fe31 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:39.650659995 +0000 UTC m=+37.149062540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls") pod "dns-default-sq9g2" (UID: "45bb5108-6076-46d7-9421-ca8ea674fe31") : secret "dns-default-metrics-tls" not found Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650696 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650700 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:39.650690272 +0000 UTC m=+37.149092804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : configmap references non-existent config key: service-ca.crt Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650734 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls podName:0e0f63bd-220f-46f4-b52c-60dccf2f6e2f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:39.650723474 +0000 UTC m=+37.149126005 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vf6rz" (UID: "0e0f63bd-220f-46f4-b52c-60dccf2f6e2f") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650607 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650750 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650765 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls podName:bd5a2cba-dca8-4af9-b0cf-c735185b0386 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:39.650755968 +0000 UTC m=+37.149158492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls") pod "cluster-samples-operator-6748489656-j7tbk" (UID: "bd5a2cba-dca8-4af9-b0cf-c735185b0386") : secret "samples-operator-tls" not found Jul 24 17:03:38.650885 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:38.650780 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:39.650770096 +0000 UTC m=+37.149172629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : secret "router-metrics-certs-default" not found Jul 24 17:03:38.882364 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.882312 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mkptd"] Jul 24 17:03:38.891426 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:38.891379 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d20e785_46d3_4a00_a8a6_50a30445dbc8.slice/crio-731965fe6bc8d7174e5fba416b2d6de976b63d48c6884c93bae14ee92a519bf6 WatchSource:0}: Error finding container 731965fe6bc8d7174e5fba416b2d6de976b63d48c6884c93bae14ee92a519bf6: Status 404 returned error can't find the container with id 731965fe6bc8d7174e5fba416b2d6de976b63d48c6884c93bae14ee92a519bf6 Jul 24 17:03:38.907359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.904421 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj"] Jul 24 17:03:38.910838 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:38.907499 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477521c4_4d67_4c6e_95b9_44dfb810a695.slice/crio-adade88bfc5407d280eb9f61a0fc38bd38fff0798a7a7327b3fbba7abd014945 WatchSource:0}: Error finding container adade88bfc5407d280eb9f61a0fc38bd38fff0798a7a7327b3fbba7abd014945: Status 404 returned error can't find the container with id adade88bfc5407d280eb9f61a0fc38bd38fff0798a7a7327b3fbba7abd014945 Jul 24 17:03:38.912349 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.911940 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg"] Jul 24 17:03:38.913521 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.913333 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-455nt"] Jul 24 17:03:38.916124 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.916070 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm"] Jul 24 17:03:38.917196 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.917179 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284"] Jul 24 17:03:38.921049 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:38.921016 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998d1e8e_6a69_47c4_a035_bbbdb44d68bc.slice/crio-b529f19154c6f5eef1f3e0c624f42922d0758f7821e3f1885962d760e93f4583 WatchSource:0}: Error finding container b529f19154c6f5eef1f3e0c624f42922d0758f7821e3f1885962d760e93f4583: Status 404 returned error can't find the container with id b529f19154c6f5eef1f3e0c624f42922d0758f7821e3f1885962d760e93f4583 Jul 24 17:03:38.922194 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:38.922133 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcd9aa9_3428_4b39_9ca7_a92c0d2ebfc0.slice/crio-a605c4f0673cfdff244ffc6d308c600dbd94e8cb7dbb5a596851fd62902882a5 WatchSource:0}: Error finding container a605c4f0673cfdff244ffc6d308c600dbd94e8cb7dbb5a596851fd62902882a5: Status 404 returned error can't find the container with id a605c4f0673cfdff244ffc6d308c600dbd94e8cb7dbb5a596851fd62902882a5 Jul 24 17:03:38.928295 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.928273 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-c5bfp"] Jul 24 17:03:38.932288 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:38.932265 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5333a0_2111_4134_b27c_8a6d49e53bc0.slice/crio-b0db06d38d528e11526e2717038424a4b5019221d038d919524edbd475527d1f WatchSource:0}: Error finding container b0db06d38d528e11526e2717038424a4b5019221d038d919524edbd475527d1f: Status 404 returned error can't find the container with id b0db06d38d528e11526e2717038424a4b5019221d038d919524edbd475527d1f Jul 24 17:03:38.958544 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.958510 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:03:38.958544 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.958524 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:03:38.961156 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.961134 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:03:38.961780 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.961139 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7d7c\"" Jul 24 17:03:38.961780 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:38.961232 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:03:39.207149 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.207109 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" event={"ID":"9d5333a0-2111-4134-b27c-8a6d49e53bc0","Type":"ContainerStarted","Data":"b0db06d38d528e11526e2717038424a4b5019221d038d919524edbd475527d1f"} Jul 24 17:03:39.209751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.209725 2552 generic.go:356] "Generic (PLEG): container finished" podID="27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8" containerID="0a82e01091cab4054e599f2a8f9d42163ba993635e00b702c859e9319fdd056a" exitCode=0 Jul 24 17:03:39.209863 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.209789 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerDied","Data":"0a82e01091cab4054e599f2a8f9d42163ba993635e00b702c859e9319fdd056a"} Jul 24 17:03:39.210969 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.210940 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-455nt" event={"ID":"3265b58a-6a78-4afa-ba20-533d76387071","Type":"ContainerStarted","Data":"2e99f02039bb9c6b767285cd29b6a53c25638bef8e0af4be00c474308eafc3e0"} Jul 24 17:03:39.212085 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.212061 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mkptd" event={"ID":"5d20e785-46d3-4a00-a8a6-50a30445dbc8","Type":"ContainerStarted","Data":"731965fe6bc8d7174e5fba416b2d6de976b63d48c6884c93bae14ee92a519bf6"} Jul 24 17:03:39.213176 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.213145 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" event={"ID":"998d1e8e-6a69-47c4-a035-bbbdb44d68bc","Type":"ContainerStarted","Data":"b529f19154c6f5eef1f3e0c624f42922d0758f7821e3f1885962d760e93f4583"} Jul 24 17:03:39.214306 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.214287 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" event={"ID":"bfa92a22-0110-4c21-8ec9-3de926b4cba3","Type":"ContainerStarted","Data":"b10cf6afc55d5c1dbd4d188e9a2f73fcc42430fd420047fb1a89cf866a41b387"} Jul 24 17:03:39.215423 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.215388 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" event={"ID":"477521c4-4d67-4c6e-95b9-44dfb810a695","Type":"ContainerStarted","Data":"adade88bfc5407d280eb9f61a0fc38bd38fff0798a7a7327b3fbba7abd014945"} Jul 24 17:03:39.216367 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.216349 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" event={"ID":"5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0","Type":"ContainerStarted","Data":"a605c4f0673cfdff244ffc6d308c600dbd94e8cb7dbb5a596851fd62902882a5"} Jul 24 17:03:39.564546 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.564449 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:39.564716 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.564652 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:03:39.564716 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.564677 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6566bdd846-59cjr: secret "image-registry-tls" not found Jul 24 17:03:39.564822 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.564750 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls podName:c4e20cf2-9271-4dfd-9143-0101267953ab nodeName:}" failed. No retries permitted until 2026-07-24 17:03:41.564727703 +0000 UTC m=+39.063130246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls") pod "image-registry-6566bdd846-59cjr" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab") : secret "image-registry-tls" not found Jul 24 17:03:39.665574 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.665535 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:39.665762 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.665586 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:39.665762 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.665635 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:39.665762 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.665667 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:39.665762 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:39.665688 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:39.665984 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.665819 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:41.665800207 +0000 UTC m=+39.164202735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : configmap references non-existent config key: service-ca.crt Jul 24 17:03:39.666203 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666180 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:39.666274 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666233 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls podName:0e0f63bd-220f-46f4-b52c-60dccf2f6e2f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:41.66622021 +0000 UTC m=+39.164622733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vf6rz" (UID: "0e0f63bd-220f-46f4-b52c-60dccf2f6e2f") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:39.666274 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666273 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:03:39.666390 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666294 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls podName:bd5a2cba-dca8-4af9-b0cf-c735185b0386 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:41.666287324 +0000 UTC m=+39.164689849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls") pod "cluster-samples-operator-6748489656-j7tbk" (UID: "bd5a2cba-dca8-4af9-b0cf-c735185b0386") : secret "samples-operator-tls" not found Jul 24 17:03:39.666390 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666329 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:03:39.666390 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666347 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:41.666341198 +0000 UTC m=+39.164743722 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : secret "router-metrics-certs-default" not found Jul 24 17:03:39.666390 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666380 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:03:39.666616 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:39.666399 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls podName:45bb5108-6076-46d7-9421-ca8ea674fe31 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:41.666393508 +0000 UTC m=+39.164796031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls") pod "dns-default-sq9g2" (UID: "45bb5108-6076-46d7-9421-ca8ea674fe31") : secret "dns-default-metrics-tls" not found Jul 24 17:03:40.234903 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:40.234861 2552 generic.go:356] "Generic (PLEG): container finished" podID="27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8" containerID="fe0d3ce5a2dc43b88d7aa12a6197e69fbf44dd320673dc476fddfc74ce4d0537" exitCode=0 Jul 24 17:03:40.235346 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:40.234944 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerDied","Data":"fe0d3ce5a2dc43b88d7aa12a6197e69fbf44dd320673dc476fddfc74ce4d0537"} Jul 24 17:03:41.246615 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.245711 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" event={"ID":"27d9e72d-ed1b-4f9f-bf79-30b1e641c9b8","Type":"ContainerStarted","Data":"14ebc61ca9ceba1749087d9021747f730802313ea29549eab34a1da5920deef9"} Jul 24 17:03:41.269318 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.269196 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rjwlv" podStartSLOduration=8.616708398 podStartE2EDuration="38.269175122s" totalImagesPullingTime="29.652466724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:41.267189434 +0000 UTC m=+38.765591979" watchObservedRunningTime="2026-07-24 17:03:41.269175122 +0000 UTC m=+38.767577649" Jul 24 17:03:41.391734 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.391671 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:41.448435 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.448377 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:41.592292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.592193 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:41.592480 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.592352 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:03:41.592480 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.592369 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6566bdd846-59cjr: secret "image-registry-tls" not found Jul 24 17:03:41.592480 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.592446 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls podName:c4e20cf2-9271-4dfd-9143-0101267953ab nodeName:}" failed. No retries permitted until 2026-07-24 17:03:45.592426125 +0000 UTC m=+43.090828670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls") pod "image-registry-6566bdd846-59cjr" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab") : secret "image-registry-tls" not found Jul 24 17:03:41.617660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.617628 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:41.671176 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.671122 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.693296 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.693346 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.693444 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693477 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.693494 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:41.693522 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693554 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls podName:0e0f63bd-220f-46f4-b52c-60dccf2f6e2f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:45.693529642 +0000 UTC m=+43.191932174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vf6rz" (UID: "0e0f63bd-220f-46f4-b52c-60dccf2f6e2f") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693570 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693620 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:45.693613545 +0000 UTC m=+43.192016069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : configmap references non-existent config key: service-ca.crt Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693661 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693675 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693706 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls podName:bd5a2cba-dca8-4af9-b0cf-c735185b0386 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:45.693690429 +0000 UTC m=+43.192092958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls") pod "cluster-samples-operator-6748489656-j7tbk" (UID: "bd5a2cba-dca8-4af9-b0cf-c735185b0386") : secret "samples-operator-tls" not found Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693760 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:45.693744406 +0000 UTC m=+43.192146935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : secret "router-metrics-certs-default" not found Jul 24 17:03:41.693751 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:41.693779 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls podName:45bb5108-6076-46d7-9421-ca8ea674fe31 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:45.693770007 +0000 UTC m=+43.192172537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls") pod "dns-default-sq9g2" (UID: "45bb5108-6076-46d7-9421-ca8ea674fe31") : secret "dns-default-metrics-tls" not found Jul 24 17:03:45.627649 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:45.627605 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:45.628096 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.627783 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:03:45.628096 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.627805 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6566bdd846-59cjr: secret "image-registry-tls" not found Jul 24 17:03:45.628096 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.627862 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls podName:c4e20cf2-9271-4dfd-9143-0101267953ab nodeName:}" failed. No retries permitted until 2026-07-24 17:03:53.627847043 +0000 UTC m=+51.126249570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls") pod "image-registry-6566bdd846-59cjr" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab") : secret "image-registry-tls" not found Jul 24 17:03:45.728199 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:45.728152 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:45.728359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:45.728245 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:45.728359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:45.728282 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:45.728359 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728326 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:03:45.728512 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728378 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:03:45.728512 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728404 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:53.728382188 +0000 UTC m=+51.226784732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : secret "router-metrics-certs-default" not found Jul 24 17:03:45.728512 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728445 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls podName:45bb5108-6076-46d7-9421-ca8ea674fe31 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:53.728429449 +0000 UTC m=+51.226831994 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls") pod "dns-default-sq9g2" (UID: "45bb5108-6076-46d7-9421-ca8ea674fe31") : secret "dns-default-metrics-tls" not found Jul 24 17:03:45.728635 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728551 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:53.728522496 +0000 UTC m=+51.226925024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : configmap references non-existent config key: service-ca.crt Jul 24 17:03:45.728635 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:45.728580 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:45.728635 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:45.728609 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:45.728777 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728696 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:45.728777 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728731 2552 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:03:45.728777 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728746 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls podName:0e0f63bd-220f-46f4-b52c-60dccf2f6e2f nodeName:}" failed. No retries permitted until 2026-07-24 17:03:53.728733034 +0000 UTC m=+51.227135571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vf6rz" (UID: "0e0f63bd-220f-46f4-b52c-60dccf2f6e2f") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:45.728866 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:45.728783 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls podName:bd5a2cba-dca8-4af9-b0cf-c735185b0386 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:53.728762853 +0000 UTC m=+51.227165380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls") pod "cluster-samples-operator-6748489656-j7tbk" (UID: "bd5a2cba-dca8-4af9-b0cf-c735185b0386") : secret "samples-operator-tls" not found Jul 24 17:03:46.222698 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:46.222655 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:46.322823 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:46.322780 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:47.266038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.265611 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" event={"ID":"bfa92a22-0110-4c21-8ec9-3de926b4cba3","Type":"ContainerStarted","Data":"0c1dc1242024641790d2deb67719f8e894b77f8785bc42997e03f05d270479b6"} Jul 24 17:03:47.267257 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.267234 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" event={"ID":"477521c4-4d67-4c6e-95b9-44dfb810a695","Type":"ContainerStarted","Data":"ba8d75f4f93f9eff418438634da17306e7ab8cc0b71790d22feed3e8ac12c917"} Jul 24 17:03:47.269903 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.269882 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" event={"ID":"5fcd9aa9-3428-4b39-9ca7-a92c0d2ebfc0","Type":"ContainerStarted","Data":"714859d228dde7314f2c105a9fbb33785a4563a3ecd4134ddd4312727276d8a9"} Jul 24 17:03:47.272043 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.272022 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/0.log" Jul 24 17:03:47.272159 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.272056 2552 generic.go:356] "Generic (PLEG): container finished" podID="9d5333a0-2111-4134-b27c-8a6d49e53bc0" containerID="41660e5df34e6e94c0f7fa49e07e560a3c246d8357a9f63fdeb17f8f0a2424c2" exitCode=255 Jul 24 17:03:47.272159 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.272143 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" event={"ID":"9d5333a0-2111-4134-b27c-8a6d49e53bc0","Type":"ContainerDied","Data":"41660e5df34e6e94c0f7fa49e07e560a3c246d8357a9f63fdeb17f8f0a2424c2"} Jul 24 17:03:47.272460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.272437 2552 scope.go:117] "RemoveContainer" containerID="41660e5df34e6e94c0f7fa49e07e560a3c246d8357a9f63fdeb17f8f0a2424c2" Jul 24 17:03:47.274347 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.273931 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-455nt" event={"ID":"3265b58a-6a78-4afa-ba20-533d76387071","Type":"ContainerStarted","Data":"e54771ef37ed274f7d3780439498c278de20740d67a9f033ad4fb468d8f169f2"} Jul 24 17:03:47.276030 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.275989 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mkptd" event={"ID":"5d20e785-46d3-4a00-a8a6-50a30445dbc8","Type":"ContainerStarted","Data":"2cadcde23e5c215e87ff31a593abae560a30c2593c5e74faef61872bad557731"} Jul 24 17:03:47.277838 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.277805 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" event={"ID":"998d1e8e-6a69-47c4-a035-bbbdb44d68bc","Type":"ContainerStarted","Data":"d62e18986b2b67635a4b82e63a3fe58f0ff58d372491de2d8a7163d19fd5fbf8"} Jul 24 17:03:47.283599 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.283554 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" podStartSLOduration=2.420980166 podStartE2EDuration="10.283538048s" totalImagesPullingTime="7.862557882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:47.281690618 +0000 UTC m=+44.780093163" watchObservedRunningTime="2026-07-24 17:03:47.283538048 +0000 UTC m=+44.781940595" Jul 24 17:03:47.296871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.296829 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" podStartSLOduration=2.428283735 podStartE2EDuration="10.296800115s" totalImagesPullingTime="7.86851638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:47.295923478 +0000 UTC m=+44.794326029" watchObservedRunningTime="2026-07-24 17:03:47.296800115 +0000 UTC m=+44.795202664" Jul 24 17:03:47.315978 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.315878 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mkptd" podStartSLOduration=2.42384392 podStartE2EDuration="10.315858779s" totalImagesPullingTime="7.892014859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:47.314864428 +0000 UTC m=+44.813266975" watchObservedRunningTime="2026-07-24 17:03:47.315858779 +0000 UTC m=+44.814261327" Jul 24 17:03:47.351543 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.351491 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-jf4vm" podStartSLOduration=2.493732549 podStartE2EDuration="10.351471335s" totalImagesPullingTime="7.857738786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:47.349474618 +0000 UTC m=+44.847877165" watchObservedRunningTime="2026-07-24 17:03:47.351471335 +0000 UTC m=+44.849873882" Jul 24 17:03:47.351943 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.351910 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-qt284" podStartSLOduration=2.496355068 podStartE2EDuration="10.351897424s" totalImagesPullingTime="7.855542356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:47.33230063 +0000 UTC m=+44.830703179" watchObservedRunningTime="2026-07-24 17:03:47.351897424 +0000 UTC m=+44.850299972" Jul 24 17:03:47.385605 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.385472 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-455nt" podStartSLOduration=2.524802557 podStartE2EDuration="10.385452438s" totalImagesPullingTime="7.860649881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:47.367524142 +0000 UTC m=+44.865926691" watchObservedRunningTime="2026-07-24 17:03:47.385452438 +0000 UTC m=+44.883854985" Jul 24 17:03:47.443051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.443016 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:47.543995 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:47.543908 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.079459 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.079356 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.141151 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.141121 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.259906 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.259832 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:48.260077 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.259947 2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:48.282470 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.282441 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:03:48.282896 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.282812 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/0.log" Jul 24 17:03:48.282896 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.282858 2552 generic.go:356] "Generic (PLEG): container finished" podID="9d5333a0-2111-4134-b27c-8a6d49e53bc0" containerID="5e983c076e480bb6b6518f166a1c11a2313150c54f7153314ce41870fe8a7c8f" exitCode=255 Jul 24 17:03:48.283008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.282938 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" event={"ID":"9d5333a0-2111-4134-b27c-8a6d49e53bc0","Type":"ContainerDied","Data":"5e983c076e480bb6b6518f166a1c11a2313150c54f7153314ce41870fe8a7c8f"} Jul 24 17:03:48.283008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.282999 2552 scope.go:117] "RemoveContainer" containerID="41660e5df34e6e94c0f7fa49e07e560a3c246d8357a9f63fdeb17f8f0a2424c2" Jul 24 17:03:48.283229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.283212 2552 scope.go:117] "RemoveContainer" containerID="5e983c076e480bb6b6518f166a1c11a2313150c54f7153314ce41870fe8a7c8f" Jul 24 17:03:48.283440 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:48.283388 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c5bfp_openshift-console-operator(9d5333a0-2111-4134-b27c-8a6d49e53bc0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" podUID="9d5333a0-2111-4134-b27c-8a6d49e53bc0" Jul 24 17:03:48.427920 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.427884 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.434604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.434578 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb"] Jul 24 17:03:48.465561 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.465528 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb"] Jul 24 17:03:48.465561 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.465544 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" Jul 24 17:03:48.468006 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.467971 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 17:03:48.468221 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.468198 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 17:03:48.468382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.468352 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-srkxv\"" Jul 24 17:03:48.526930 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.526885 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.553120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.553092 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4459v\" (UniqueName: \"kubernetes.io/projected/ecd808f2-451a-4b8a-a0df-0461eef9e6b5-kube-api-access-4459v\") pod \"migrator-6865c9967-gjfqb\" (UID: \"ecd808f2-451a-4b8a-a0df-0461eef9e6b5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" Jul 24 17:03:48.626961 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.626929 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.654323 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.654293 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4459v\" (UniqueName: \"kubernetes.io/projected/ecd808f2-451a-4b8a-a0df-0461eef9e6b5-kube-api-access-4459v\") pod \"migrator-6865c9967-gjfqb\" (UID: \"ecd808f2-451a-4b8a-a0df-0461eef9e6b5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" Jul 24 17:03:48.664038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.664012 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4459v\" (UniqueName: \"kubernetes.io/projected/ecd808f2-451a-4b8a-a0df-0461eef9e6b5-kube-api-access-4459v\") pod \"migrator-6865c9967-gjfqb\" (UID: \"ecd808f2-451a-4b8a-a0df-0461eef9e6b5\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" Jul 24 17:03:48.776099 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.775996 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" Jul 24 17:03:48.785924 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.785895 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.888236 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.887976 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:48.932287 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.932195 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb"] Jul 24 17:03:48.935940 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:48.935911 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd808f2_451a_4b8a_a0df_0461eef9e6b5.slice/crio-f7382d63e4c1beaccb12c6e97c546d6dad85fd36cc03ac4c2791f3b2cd306751 WatchSource:0}: Error finding container f7382d63e4c1beaccb12c6e97c546d6dad85fd36cc03ac4c2791f3b2cd306751: Status 404 returned error can't find the container with id f7382d63e4c1beaccb12c6e97c546d6dad85fd36cc03ac4c2791f3b2cd306751 Jul 24 17:03:48.988734 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:48.988701 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dg95g"] Jul 24 17:03:49.017977 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.017943 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:49.022473 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.022445 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dg95g"] Jul 24 17:03:49.022597 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.022541 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.025200 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.025173 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 17:03:49.025325 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.025227 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 17:03:49.025325 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.025234 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bh4kl\"" Jul 24 17:03:49.057697 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.057671 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86216dce-69aa-4dc1-9231-85ad8505db50-crio-socket\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.057810 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.057728 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.057810 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.057785 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.057926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.057811 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86216dce-69aa-4dc1-9231-85ad8505db50-data-volume\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.057926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.057882 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ml9\" (UniqueName: \"kubernetes.io/projected/86216dce-69aa-4dc1-9231-85ad8505db50-kube-api-access-r7ml9\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.094245 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.094211 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:49.158539 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.158511 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.158691 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.158555 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.158691 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.158579 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86216dce-69aa-4dc1-9231-85ad8505db50-data-volume\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.158691 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.158604 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ml9\" (UniqueName: \"kubernetes.io/projected/86216dce-69aa-4dc1-9231-85ad8505db50-kube-api-access-r7ml9\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.158691 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.158642 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:03:49.158691 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.158666 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:03:49.158691 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.158689 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86216dce-69aa-4dc1-9231-85ad8505db50-crio-socket\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.158990 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.158713 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:49.65869315 +0000 UTC m=+47.157095681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : secret "insights-runtime-extractor-tls" not found Jul 24 17:03:49.158990 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.158733 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:49.658724925 +0000 UTC m=+47.157127456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : configmap "kube-rbac-proxy" not found Jul 24 17:03:49.158990 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.158744 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/86216dce-69aa-4dc1-9231-85ad8505db50-crio-socket\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.159111 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.158989 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/86216dce-69aa-4dc1-9231-85ad8505db50-data-volume\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.169374 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.169354 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ml9\" (UniqueName: \"kubernetes.io/projected/86216dce-69aa-4dc1-9231-85ad8505db50-kube-api-access-r7ml9\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.287911 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.287830 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:03:49.288361 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.288195 2552 scope.go:117] "RemoveContainer" containerID="5e983c076e480bb6b6518f166a1c11a2313150c54f7153314ce41870fe8a7c8f" Jul 24 17:03:49.288444 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.288385 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-c5bfp_openshift-console-operator(9d5333a0-2111-4134-b27c-8a6d49e53bc0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" podUID="9d5333a0-2111-4134-b27c-8a6d49e53bc0" Jul 24 17:03:49.288994 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.288972 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" event={"ID":"ecd808f2-451a-4b8a-a0df-0461eef9e6b5","Type":"ContainerStarted","Data":"f7382d63e4c1beaccb12c6e97c546d6dad85fd36cc03ac4c2791f3b2cd306751"} Jul 24 17:03:49.663922 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.663871 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.664113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:49.663946 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:49.664113 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.664029 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:03:49.664113 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.664092 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:50.664076968 +0000 UTC m=+48.162479492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : secret "insights-runtime-extractor-tls" not found Jul 24 17:03:49.664113 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.664074 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:03:49.664284 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:49.664157 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:50.664143046 +0000 UTC m=+48.162545574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : configmap "kube-rbac-proxy" not found Jul 24 17:03:50.383568 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.383535 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m6grh"] Jul 24 17:03:50.412623 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.412594 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m6grh"] Jul 24 17:03:50.412762 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.412691 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.415323 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.415302 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 17:03:50.416396 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.416371 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 17:03:50.416505 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.416375 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-8s7kz\"" Jul 24 17:03:50.416505 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.416376 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 17:03:50.416505 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.416383 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 17:03:50.420946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.420922 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:50.472264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.472234 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95c4a4f1-7349-4151-8cf7-2d38fca9b341-signing-key\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.472399 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.472318 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjhc9\" (UniqueName: \"kubernetes.io/projected/95c4a4f1-7349-4151-8cf7-2d38fca9b341-kube-api-access-rjhc9\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.472486 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.472467 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95c4a4f1-7349-4151-8cf7-2d38fca9b341-signing-cabundle\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.525788 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.525758 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:50.573097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.573070 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95c4a4f1-7349-4151-8cf7-2d38fca9b341-signing-cabundle\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.573224 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.573140 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95c4a4f1-7349-4151-8cf7-2d38fca9b341-signing-key\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.573224 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.573213 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjhc9\" (UniqueName: \"kubernetes.io/projected/95c4a4f1-7349-4151-8cf7-2d38fca9b341-kube-api-access-rjhc9\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.573770 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.573752 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95c4a4f1-7349-4151-8cf7-2d38fca9b341-signing-cabundle\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.575456 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.575429 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95c4a4f1-7349-4151-8cf7-2d38fca9b341-signing-key\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.580974 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.580953 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjhc9\" (UniqueName: \"kubernetes.io/projected/95c4a4f1-7349-4151-8cf7-2d38fca9b341-kube-api-access-rjhc9\") pod \"service-ca-658d7b7c78-m6grh\" (UID: \"95c4a4f1-7349-4151-8cf7-2d38fca9b341\") " pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.674308 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.674277 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:50.674493 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.674323 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:50.674493 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:50.674399 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:03:50.674493 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:50.674441 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:03:50.674493 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:50.674474 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:52.674458927 +0000 UTC m=+50.172861457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : configmap "kube-rbac-proxy" not found Jul 24 17:03:50.674627 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:50.674503 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:52.674489733 +0000 UTC m=+50.172892262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : secret "insights-runtime-extractor-tls" not found Jul 24 17:03:50.721953 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.721926 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" Jul 24 17:03:50.875371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:50.875342 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-m6grh"] Jul 24 17:03:50.878891 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:03:50.878867 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c4a4f1_7349_4151_8cf7_2d38fca9b341.slice/crio-bca510b6d4682238daa016ab015d44473a698f3cba3ccffe16a97000d3bc4c95 WatchSource:0}: Error finding container bca510b6d4682238daa016ab015d44473a698f3cba3ccffe16a97000d3bc4c95: Status 404 returned error can't find the container with id bca510b6d4682238daa016ab015d44473a698f3cba3ccffe16a97000d3bc4c95 Jul 24 17:03:51.178957 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.178887 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgptf_c68158e9-2ffc-401e-a9cd-5eb60b63d7c3/dns-node-resolver/0.log" Jul 24 17:03:51.296985 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.296945 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" event={"ID":"ecd808f2-451a-4b8a-a0df-0461eef9e6b5","Type":"ContainerStarted","Data":"a2b15e8d831fc678ee63e1318ece296f01ed598ff607ef0a569d909464dbd554"} Jul 24 17:03:51.297172 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.297006 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" event={"ID":"ecd808f2-451a-4b8a-a0df-0461eef9e6b5","Type":"ContainerStarted","Data":"20a10aad1139b04355fb1166c0c402efaa5a8175ce3ca8a9ada955a37de1f85c"} Jul 24 17:03:51.298462 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.298434 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" event={"ID":"95c4a4f1-7349-4151-8cf7-2d38fca9b341","Type":"ContainerStarted","Data":"bcdb7d6cc704f5bd33a88463ec6dd44045313b54041aec4199e875b0fd6c247c"} Jul 24 17:03:51.298571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.298475 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" event={"ID":"95c4a4f1-7349-4151-8cf7-2d38fca9b341","Type":"ContainerStarted","Data":"bca510b6d4682238daa016ab015d44473a698f3cba3ccffe16a97000d3bc4c95"} Jul 24 17:03:51.316778 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.316707 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gjfqb" podStartSLOduration=1.568082626 podStartE2EDuration="3.316688648s" totalImagesPullingTime="1.748606022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:51.31549936 +0000 UTC m=+48.813901905" watchObservedRunningTime="2026-07-24 17:03:51.316688648 +0000 UTC m=+48.815091194" Jul 24 17:03:51.321391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.321364 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:51.345827 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.345780 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-m6grh" podStartSLOduration=1.34576648 podStartE2EDuration="1.34576648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:51.345186624 +0000 UTC m=+48.843589171" watchObservedRunningTime="2026-07-24 17:03:51.34576648 +0000 UTC m=+48.844169058" Jul 24 17:03:51.706156 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.706112 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:51.884079 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.884038 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:51.980556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.980472 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fbnqh_d239a832-beca-4842-b38f-fee778ed3b0d/node-ca/0.log" Jul 24 17:03:51.981052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:51.981016 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:52.067907 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:52.067870 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:52.578831 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:52.578790 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mkptd_5d20e785-46d3-4a00-a8a6-50a30445dbc8/serve-healthcheck-canary/0.log" Jul 24 17:03:52.618723 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:52.618689 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:52.692219 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:52.692183 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:52.692374 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:52.692231 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:52.692374 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:52.692309 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:03:52.692374 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:52.692333 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:03:52.692374 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:52.692367 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:56.692350357 +0000 UTC m=+54.190752884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : configmap "kube-rbac-proxy" not found Jul 24 17:03:52.692535 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:52.692391 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:03:56.692377831 +0000 UTC m=+54.190780359 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : secret "insights-runtime-extractor-tls" not found Jul 24 17:03:52.778750 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:52.778718 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gjfqb_ecd808f2-451a-4b8a-a0df-0461eef9e6b5/migrator/0.log" Jul 24 17:03:52.978783 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:52.978762 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gjfqb_ecd808f2-451a-4b8a-a0df-0461eef9e6b5/graceful-termination/0.log" Jul 24 17:03:53.179943 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.179876 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg_bfa92a22-0110-4c21-8ec9-3de926b4cba3/kube-storage-version-migrator-operator/0.log" Jul 24 17:03:53.427837 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.427791 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:53.701663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.701584 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:03:53.701814 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.701736 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:03:53.701814 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.701756 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6566bdd846-59cjr: secret "image-registry-tls" not found Jul 24 17:03:53.701814 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.701808 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls podName:c4e20cf2-9271-4dfd-9143-0101267953ab nodeName:}" failed. No retries permitted until 2026-07-24 17:04:09.701794254 +0000 UTC m=+67.200196778 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls") pod "image-registry-6566bdd846-59cjr" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab") : secret "image-registry-tls" not found Jul 24 17:03:53.802077 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.802042 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.802088 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.802146 2552 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.802158 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.802201 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.802228 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.802277 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls podName:0e0f63bd-220f-46f4-b52c-60dccf2f6e2f nodeName:}" failed. No retries permitted until 2026-07-24 17:04:09.802244761 +0000 UTC m=+67.300647297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vf6rz" (UID: "0e0f63bd-220f-46f4-b52c-60dccf2f6e2f") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.802325 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:09.802311135 +0000 UTC m=+67.300713665 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : configmap references non-existent config key: service-ca.crt Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.802336 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:03:53.802585 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.802389 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls podName:45bb5108-6076-46d7-9421-ca8ea674fe31 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:09.802374494 +0000 UTC m=+67.300777018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls") pod "dns-default-sq9g2" (UID: "45bb5108-6076-46d7-9421-ca8ea674fe31") : secret "dns-default-metrics-tls" not found Jul 24 17:03:53.803088 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.802628 2552 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:03:53.803088 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:53.802674 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs podName:69009bfd-dd6d-4838-9981-c3b080dbadd6 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:09.802660836 +0000 UTC m=+67.301063365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs") pod "router-default-7d8cbf98d5-bg4mh" (UID: "69009bfd-dd6d-4838-9981-c3b080dbadd6") : secret "router-metrics-certs-default" not found Jul 24 17:03:53.803088 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.802913 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:53.804675 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.804644 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd5a2cba-dca8-4af9-b0cf-c735185b0386-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-j7tbk\" (UID: \"bd5a2cba-dca8-4af9-b0cf-c735185b0386\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:53.812108 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.812088 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" Jul 24 17:03:53.903060 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.903026 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:53.955536 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:53.955511 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk"] Jul 24 17:03:54.004204 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:54.004160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:54.198604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:54.198558 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:54.312719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:54.312640 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" event={"ID":"bd5a2cba-dca8-4af9-b0cf-c735185b0386","Type":"ContainerStarted","Data":"9546fa97a67cb5a64be97f41ea6805095e99ff696d832e3334800ba794ae6446"} Jul 24 17:03:54.526507 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:54.526469 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:54.626691 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:54.626614 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:55.401800 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:55.401723 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:55.502101 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:55.502068 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:55.604785 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:55.604753 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:56.030194 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.030159 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:56.201545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.201509 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:56.310514 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.310480 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:56.320625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.320596 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" event={"ID":"bd5a2cba-dca8-4af9-b0cf-c735185b0386","Type":"ContainerStarted","Data":"09298a1dcdd59dd0f5f18a6ed72465b47dc467572342ba1bb2dd5d8ba1c25c0f"} Jul 24 17:03:56.728821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.728782 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:56.729324 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.728838 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:56.729324 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:56.728931 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:03:56.729324 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:03:56.728997 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls podName:86216dce-69aa-4dc1-9231-85ad8505db50 nodeName:}" failed. No retries permitted until 2026-07-24 17:04:04.728982657 +0000 UTC m=+62.227385181 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dg95g" (UID: "86216dce-69aa-4dc1-9231-85ad8505db50") : secret "insights-runtime-extractor-tls" not found Jul 24 17:03:56.729533 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.729447 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/86216dce-69aa-4dc1-9231-85ad8505db50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:03:56.799468 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:56.799431 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:57.006625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:57.006531 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:57.325564 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:57.325476 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" event={"ID":"bd5a2cba-dca8-4af9-b0cf-c735185b0386","Type":"ContainerStarted","Data":"836783f62f119254361ae835a5e5876d5c9953d4ef751d28ed2a8371d4a621e5"} Jul 24 17:03:57.340932 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:57.340889 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-j7tbk" podStartSLOduration=18.138481146 podStartE2EDuration="20.340873843s" totalImagesPullingTime="2.202392697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:57.340275236 +0000 UTC m=+54.838677783" watchObservedRunningTime="2026-07-24 17:03:57.340873843 +0000 UTC m=+54.839276390" Jul 24 17:03:57.605688 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:57.605597 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:57.922098 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:57.922066 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:58.062605 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.062567 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:58.204850 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.204762 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:58.259655 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.259619 2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:58.259815 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.259666 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:58.260004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.259992 2552 scope.go:117] "RemoveContainer" containerID="5e983c076e480bb6b6518f166a1c11a2313150c54f7153314ce41870fe8a7c8f" Jul 24 17:03:58.331188 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.331166 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:03:58.331315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.331286 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" event={"ID":"9d5333a0-2111-4134-b27c-8a6d49e53bc0","Type":"ContainerStarted","Data":"0df4af4fc3ab28fc3f0a98e879ba6f5ccd481d799bfb6fe123fe1fa933471588"} Jul 24 17:03:58.331754 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.331734 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:58.333328 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.333296 2552 patch_prober.go:28] interesting pod/console-operator-584dcb6b54-c5bfp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body= Jul 24 17:03:58.333454 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.333339 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" podUID="9d5333a0-2111-4134-b27c-8a6d49e53bc0" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.16:8443/readyz\": dial tcp 10.132.0.16:8443: connect: connection refused" Jul 24 17:03:58.348142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.348095 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" podStartSLOduration=13.502603365 podStartE2EDuration="21.348076606s" totalImagesPullingTime="7.845473241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:03:58.346956149 +0000 UTC m=+55.845358696" watchObservedRunningTime="2026-07-24 17:03:58.348076606 +0000 UTC m=+55.846479156" Jul 24 17:03:58.602607 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:58.602527 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.008844 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.008803 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.144855 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.144813 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.245760 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.245727 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.339035 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.338944 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-c5bfp" Jul 24 17:03:59.345307 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.345283 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.441541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.441510 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.620442 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.620341 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.806161 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.806123 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:03:59.909154 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:03:59.909127 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:00.009156 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:00.009116 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:00.066903 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:00.066874 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:00.206384 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:00.206303 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:00.448934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:00.448907 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:00.504945 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:00.504841 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:00.599094 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:00.599062 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:01.028023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:01.027991 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:01.211315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:01.211281 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:01.311496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:01.311404 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:01.404887 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:01.404859 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:02.010543 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.010511 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:02.198517 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.198490 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gd69m" Jul 24 17:04:02.256475 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.256400 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:02.317983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.317907 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:02.419650 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.419614 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:02.613163 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.613077 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:02.810023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.809986 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:02.991630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:02.991594 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:04.803765 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:04.803733 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:04:04.806199 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:04.806167 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/86216dce-69aa-4dc1-9231-85ad8505db50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dg95g\" (UID: \"86216dce-69aa-4dc1-9231-85ad8505db50\") " pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:04:04.933701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:04.933668 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bh4kl\"" Jul 24 17:04:04.941823 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:04.941795 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dg95g" Jul 24 17:04:05.065514 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:05.065355 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dg95g"] Jul 24 17:04:05.067945 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:05.067908 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86216dce_69aa_4dc1_9231_85ad8505db50.slice/crio-16d13e5f42312482d3cf24e14a2fcccc05671b6efdec76fc6c223af539fce9aa WatchSource:0}: Error finding container 16d13e5f42312482d3cf24e14a2fcccc05671b6efdec76fc6c223af539fce9aa: Status 404 returned error can't find the container with id 16d13e5f42312482d3cf24e14a2fcccc05671b6efdec76fc6c223af539fce9aa Jul 24 17:04:05.354882 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:05.354796 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg95g" event={"ID":"86216dce-69aa-4dc1-9231-85ad8505db50","Type":"ContainerStarted","Data":"16d13e5f42312482d3cf24e14a2fcccc05671b6efdec76fc6c223af539fce9aa"} Jul 24 17:04:08.642786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.642745 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:04:08.645351 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.645319 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:04:08.655683 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.655656 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/895d8902-9011-45c2-8273-5beb236a0a5a-metrics-certs\") pod \"network-metrics-daemon-w8pxz\" (UID: \"895d8902-9011-45c2-8273-5beb236a0a5a\") " pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:04:08.676618 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.676594 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7d7c\"" Jul 24 17:04:08.684707 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.684683 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8pxz" Jul 24 17:04:08.743316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.743265 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:04:08.743682 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.743327 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:04:08.745988 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.745953 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:04:08.746377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.746348 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhk6\" (UniqueName: \"kubernetes.io/projected/37f2b5e7-f1cc-4283-b72d-e75f6d253b2b-kube-api-access-6xhk6\") pod \"network-check-target-586fh\" (UID: \"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b\") " pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:04:08.755839 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.755809 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4985b5b-1bdb-4bd8-b250-67e63f89d01f-original-pull-secret\") pod \"global-pull-secret-syncer-zmnnx\" (UID: \"b4985b5b-1bdb-4bd8-b250-67e63f89d01f\") " pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:04:08.870525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.870489 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lpvrb\"" Jul 24 17:04:08.878569 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.878538 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:04:08.969392 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:08.969363 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmnnx" Jul 24 17:04:09.753465 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.753427 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:04:09.755966 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.755942 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"image-registry-6566bdd846-59cjr\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:04:09.854564 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.854530 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:09.854719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.854577 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:04:09.854719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.854598 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:09.854719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.854637 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:04:09.855268 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.855246 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69009bfd-dd6d-4838-9981-c3b080dbadd6-service-ca-bundle\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:09.856923 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.856898 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/45bb5108-6076-46d7-9421-ca8ea674fe31-metrics-tls\") pod \"dns-default-sq9g2\" (UID: \"45bb5108-6076-46d7-9421-ca8ea674fe31\") " pod="openshift-dns/dns-default-sq9g2" Jul 24 17:04:09.857057 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.856964 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e0f63bd-220f-46f4-b52c-60dccf2f6e2f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vf6rz\" (UID: \"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:04:09.857118 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.857069 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69009bfd-dd6d-4838-9981-c3b080dbadd6-metrics-certs\") pod \"router-default-7d8cbf98d5-bg4mh\" (UID: \"69009bfd-dd6d-4838-9981-c3b080dbadd6\") " pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:09.972324 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.972300 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v9fwg\"" Jul 24 17:04:09.980184 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.980162 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:04:09.984089 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.984067 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9tk6v\"" Jul 24 17:04:09.991829 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:09.991603 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:10.007820 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.007757 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gnbfc\"" Jul 24 17:04:10.015851 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.015815 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sq9g2" Jul 24 17:04:10.016190 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.016168 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-586fh"] Jul 24 17:04:10.020077 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:10.020019 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f2b5e7_f1cc_4283_b72d_e75f6d253b2b.slice/crio-36f3bbebf50c1ee12243734a65d240885752d86c9c7d47377f244054bb3851d5 WatchSource:0}: Error finding container 36f3bbebf50c1ee12243734a65d240885752d86c9c7d47377f244054bb3851d5: Status 404 returned error can't find the container with id 36f3bbebf50c1ee12243734a65d240885752d86c9c7d47377f244054bb3851d5 Jul 24 17:04:10.034343 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.034311 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6d47d\"" Jul 24 17:04:10.042269 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.042243 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" Jul 24 17:04:10.176015 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.175986 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6566bdd846-59cjr"] Jul 24 17:04:10.185111 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.185083 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d8cbf98d5-bg4mh"] Jul 24 17:04:10.221808 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.221782 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sq9g2"] Jul 24 17:04:10.233558 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:10.233528 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e20cf2_9271_4dfd_9143_0101267953ab.slice/crio-702d2f7abc802fc3fa71debdc39cacacca2ad257614266e94a3773c2224689ba WatchSource:0}: Error finding container 702d2f7abc802fc3fa71debdc39cacacca2ad257614266e94a3773c2224689ba: Status 404 returned error can't find the container with id 702d2f7abc802fc3fa71debdc39cacacca2ad257614266e94a3773c2224689ba Jul 24 17:04:10.234176 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:10.234148 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69009bfd_dd6d_4838_9981_c3b080dbadd6.slice/crio-b7e4e7377f2e5c1beb5a98d4ab6ed36951effff5de79827bff354ebc824c5575 WatchSource:0}: Error finding container b7e4e7377f2e5c1beb5a98d4ab6ed36951effff5de79827bff354ebc824c5575: Status 404 returned error can't find the container with id b7e4e7377f2e5c1beb5a98d4ab6ed36951effff5de79827bff354ebc824c5575 Jul 24 17:04:10.235088 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:10.234973 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45bb5108_6076_46d7_9421_ca8ea674fe31.slice/crio-dc1876b9dfe7c494e3d81f3648ccfd8f8d26dd1cb423d9110ad91a50af585754 WatchSource:0}: Error finding container dc1876b9dfe7c494e3d81f3648ccfd8f8d26dd1cb423d9110ad91a50af585754: Status 404 returned error can't find the container with id dc1876b9dfe7c494e3d81f3648ccfd8f8d26dd1cb423d9110ad91a50af585754 Jul 24 17:04:10.243783 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.243757 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8pxz"] Jul 24 17:04:10.244710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.244694 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmnnx"] Jul 24 17:04:10.246203 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:10.246181 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895d8902_9011_45c2_8273_5beb236a0a5a.slice/crio-84cf53fab818df9264ed52f2dcd974b74ca18cacf7d7adad8875e364db288e5f WatchSource:0}: Error finding container 84cf53fab818df9264ed52f2dcd974b74ca18cacf7d7adad8875e364db288e5f: Status 404 returned error can't find the container with id 84cf53fab818df9264ed52f2dcd974b74ca18cacf7d7adad8875e364db288e5f Jul 24 17:04:10.246838 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:10.246813 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4985b5b_1bdb_4bd8_b250_67e63f89d01f.slice/crio-a86abac87a0af46947cf791327d47ede2dfd64189366303e71851b90ee510bad WatchSource:0}: Error finding container a86abac87a0af46947cf791327d47ede2dfd64189366303e71851b90ee510bad: Status 404 returned error can't find the container with id a86abac87a0af46947cf791327d47ede2dfd64189366303e71851b90ee510bad Jul 24 17:04:10.251174 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.251133 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:10.303764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.303734 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:10.374402 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.374368 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" event={"ID":"69009bfd-dd6d-4838-9981-c3b080dbadd6","Type":"ContainerStarted","Data":"555d53b95a91c9f184d8236c73ae2633d6cfcf49fb61e02e499cc9cd2ce83499"} Jul 24 17:04:10.374549 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.374483 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" event={"ID":"69009bfd-dd6d-4838-9981-c3b080dbadd6","Type":"ContainerStarted","Data":"b7e4e7377f2e5c1beb5a98d4ab6ed36951effff5de79827bff354ebc824c5575"} Jul 24 17:04:10.375619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.375551 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" event={"ID":"c4e20cf2-9271-4dfd-9143-0101267953ab","Type":"ContainerStarted","Data":"702d2f7abc802fc3fa71debdc39cacacca2ad257614266e94a3773c2224689ba"} Jul 24 17:04:10.377303 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.377281 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8pxz" event={"ID":"895d8902-9011-45c2-8273-5beb236a0a5a","Type":"ContainerStarted","Data":"84cf53fab818df9264ed52f2dcd974b74ca18cacf7d7adad8875e364db288e5f"} Jul 24 17:04:10.378479 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.378432 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sq9g2" event={"ID":"45bb5108-6076-46d7-9421-ca8ea674fe31","Type":"ContainerStarted","Data":"dc1876b9dfe7c494e3d81f3648ccfd8f8d26dd1cb423d9110ad91a50af585754"} Jul 24 17:04:10.379802 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.379775 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmnnx" event={"ID":"b4985b5b-1bdb-4bd8-b250-67e63f89d01f","Type":"ContainerStarted","Data":"a86abac87a0af46947cf791327d47ede2dfd64189366303e71851b90ee510bad"} Jul 24 17:04:10.381200 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.381168 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-586fh" event={"ID":"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b","Type":"ContainerStarted","Data":"1c8d2e0a94109ed2fcac1c4627f18f12f04c8a7accc8e542047009bfabdc247f"} Jul 24 17:04:10.381296 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.381218 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-586fh" event={"ID":"37f2b5e7-f1cc-4283-b72d-e75f6d253b2b","Type":"ContainerStarted","Data":"36f3bbebf50c1ee12243734a65d240885752d86c9c7d47377f244054bb3851d5"} Jul 24 17:04:10.381353 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.381307 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:04:10.391783 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.391759 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz"] Jul 24 17:04:10.394323 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:10.394300 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0f63bd_220f_46f4_b52c_60dccf2f6e2f.slice/crio-6f4740ff7814305492a3da7e95ed51791becd5f9a1e8dc284c726e5f6620d47b WatchSource:0}: Error finding container 6f4740ff7814305492a3da7e95ed51791becd5f9a1e8dc284c726e5f6620d47b: Status 404 returned error can't find the container with id 6f4740ff7814305492a3da7e95ed51791becd5f9a1e8dc284c726e5f6620d47b Jul 24 17:04:10.397856 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.397816 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-586fh" podStartSLOduration=67.397804211 podStartE2EDuration="1m7.397804211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:10.397196421 +0000 UTC m=+67.895598969" watchObservedRunningTime="2026-07-24 17:04:10.397804211 +0000 UTC m=+67.896206757" Jul 24 17:04:10.639229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.639144 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:10.741785 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:10.741671 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:11.304212 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.302963 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc"] Jul 24 17:04:11.329480 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.328211 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b"] Jul 24 17:04:11.329657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.329634 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.331879 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.331852 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 17:04:11.332039 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.331886 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 17:04:11.332369 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.332209 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 17:04:11.333591 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.333569 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 17:04:11.344109 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.344086 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn"] Jul 24 17:04:11.344528 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.344497 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.347049 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.347009 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-n9c47\"" Jul 24 17:04:11.347386 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.347368 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 17:04:11.347765 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.347748 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 17:04:11.359566 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.359543 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b"] Jul 24 17:04:11.359676 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.359590 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc"] Jul 24 17:04:11.359733 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.359684 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.360078 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.360057 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm"] Jul 24 17:04:11.363491 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.362730 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 17:04:11.363491 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.362948 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 17:04:11.364329 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.364095 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 17:04:11.364329 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.364156 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 17:04:11.371659 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.370707 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn"] Jul 24 17:04:11.371659 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.370745 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm"] Jul 24 17:04:11.371659 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.370828 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.373603 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.373371 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-b84gl\"" Jul 24 17:04:11.373603 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.373402 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 17:04:11.384958 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.384796 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:11.390031 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.389996 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg95g" event={"ID":"86216dce-69aa-4dc1-9231-85ad8505db50","Type":"ContainerStarted","Data":"fe8ce6bd0733c5a0a067aba18f55022bc71b6a2dfe9273170435bd536ece1063"} Jul 24 17:04:11.393516 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.393448 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" event={"ID":"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f","Type":"ContainerStarted","Data":"6f4740ff7814305492a3da7e95ed51791becd5f9a1e8dc284c726e5f6620d47b"} Jul 24 17:04:11.396857 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.396744 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" event={"ID":"c4e20cf2-9271-4dfd-9143-0101267953ab","Type":"ContainerStarted","Data":"0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6"} Jul 24 17:04:11.397297 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.397260 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:04:11.403130 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.403111 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-gr78g"] Jul 24 17:04:11.426767 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.426714 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-gr78g"] Jul 24 17:04:11.426922 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.426829 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-gr78g" Jul 24 17:04:11.429985 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.429961 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 17:04:11.431894 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.431875 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hk2vj\"" Jul 24 17:04:11.432147 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.431935 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 17:04:11.459848 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.459098 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" podStartSLOduration=34.459076792 podStartE2EDuration="34.459076792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:11.45706143 +0000 UTC m=+68.955463977" watchObservedRunningTime="2026-07-24 17:04:11.459076792 +0000 UTC m=+68.957479340" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470402 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b95cb83-2ecb-47a0-97f7-90b8a8b0b981-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-zsp7b\" (UID: \"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470533 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ft9k\" (UniqueName: \"kubernetes.io/projected/b40c90cb-7c97-488e-83b0-b243919bbec3-kube-api-access-4ft9k\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470563 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b95cb83-2ecb-47a0-97f7-90b8a8b0b981-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-zsp7b\" (UID: \"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470611 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470639 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm\" (UID: \"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470677 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1b26b2d-e39a-468f-8bb9-12a99af5881b-tmp\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470704 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtst\" (UniqueName: \"kubernetes.io/projected/b1b26b2d-e39a-468f-8bb9-12a99af5881b-kube-api-access-7xtst\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470742 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-ca\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470776 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470814 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-hub\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470860 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4lj\" (UniqueName: \"kubernetes.io/projected/c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5-kube-api-access-bs4lj\") pod \"managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm\" (UID: \"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470903 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1b26b2d-e39a-468f-8bb9-12a99af5881b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.471120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.470932 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c90cb-7c97-488e-83b0-b243919bbec3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.508182 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.507876 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" podStartSLOduration=68.507853973 podStartE2EDuration="1m8.507853973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:11.506023828 +0000 UTC m=+69.004426378" watchObservedRunningTime="2026-07-24 17:04:11.507853973 +0000 UTC m=+69.006256499" Jul 24 17:04:11.513571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.513509 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.571652 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-ca\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.571696 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.571762 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-hub\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.571817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4lj\" (UniqueName: \"kubernetes.io/projected/c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5-kube-api-access-bs4lj\") pod \"managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm\" (UID: \"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.571870 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1b26b2d-e39a-468f-8bb9-12a99af5881b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.571915 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c90cb-7c97-488e-83b0-b243919bbec3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.571969 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5cp\" (UniqueName: \"kubernetes.io/projected/c1e023d0-0c62-4c0e-a105-c4c18106d90e-kube-api-access-fd5cp\") pod \"downloads-7c97dc4555-gr78g\" (UID: \"c1e023d0-0c62-4c0e-a105-c4c18106d90e\") " pod="openshift-console/downloads-7c97dc4555-gr78g" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.572026 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b95cb83-2ecb-47a0-97f7-90b8a8b0b981-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-zsp7b\" (UID: \"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.572080 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ft9k\" (UniqueName: \"kubernetes.io/projected/b40c90cb-7c97-488e-83b0-b243919bbec3-kube-api-access-4ft9k\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.572107 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b95cb83-2ecb-47a0-97f7-90b8a8b0b981-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-zsp7b\" (UID: \"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.572183 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.572207 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm\" (UID: \"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.572258 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1b26b2d-e39a-468f-8bb9-12a99af5881b-tmp\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.574769 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.572286 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtst\" (UniqueName: \"kubernetes.io/projected/b1b26b2d-e39a-468f-8bb9-12a99af5881b-kube-api-access-7xtst\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.576484 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.576347 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b40c90cb-7c97-488e-83b0-b243919bbec3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.577275 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.576988 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1b26b2d-e39a-468f-8bb9-12a99af5881b-tmp\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.577395 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.577316 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b95cb83-2ecb-47a0-97f7-90b8a8b0b981-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-zsp7b\" (UID: \"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.579004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.578496 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm\" (UID: \"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.580336 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.579891 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.580336 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.580287 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1b26b2d-e39a-468f-8bb9-12a99af5881b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.581892 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.581868 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.582322 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.582301 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-hub\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.583155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.582594 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtst\" (UniqueName: \"kubernetes.io/projected/b1b26b2d-e39a-468f-8bb9-12a99af5881b-kube-api-access-7xtst\") pod \"klusterlet-addon-workmgr-7ddd47b78d-gtfvc\" (UID: \"b1b26b2d-e39a-468f-8bb9-12a99af5881b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.583155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.583116 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4b95cb83-2ecb-47a0-97f7-90b8a8b0b981-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-zsp7b\" (UID: \"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.584315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.584277 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b40c90cb-7c97-488e-83b0-b243919bbec3-ca\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.586637 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.586588 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ft9k\" (UniqueName: \"kubernetes.io/projected/b40c90cb-7c97-488e-83b0-b243919bbec3-kube-api-access-4ft9k\") pod \"cluster-proxy-proxy-agent-7b6dccd775-4p7pn\" (UID: \"b40c90cb-7c97-488e-83b0-b243919bbec3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.589521 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.589484 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4lj\" (UniqueName: \"kubernetes.io/projected/c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5-kube-api-access-bs4lj\") pod \"managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm\" (UID: \"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.648671 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.648392 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:11.660508 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.660377 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" Jul 24 17:04:11.673106 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.673034 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5cp\" (UniqueName: \"kubernetes.io/projected/c1e023d0-0c62-4c0e-a105-c4c18106d90e-kube-api-access-fd5cp\") pod \"downloads-7c97dc4555-gr78g\" (UID: \"c1e023d0-0c62-4c0e-a105-c4c18106d90e\") " pod="openshift-console/downloads-7c97dc4555-gr78g" Jul 24 17:04:11.684005 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.683805 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" Jul 24 17:04:11.694299 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.693696 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5cp\" (UniqueName: \"kubernetes.io/projected/c1e023d0-0c62-4c0e-a105-c4c18106d90e-kube-api-access-fd5cp\") pod \"downloads-7c97dc4555-gr78g\" (UID: \"c1e023d0-0c62-4c0e-a105-c4c18106d90e\") " pod="openshift-console/downloads-7c97dc4555-gr78g" Jul 24 17:04:11.694299 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.694199 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:11.695441 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.695368 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" Jul 24 17:04:11.748605 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.747389 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-gr78g" Jul 24 17:04:11.791154 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.791053 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:11.907434 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.907364 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b"] Jul 24 17:04:11.908649 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.908623 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc"] Jul 24 17:04:11.934982 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.934952 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm"] Jul 24 17:04:11.960771 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.960660 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn"] Jul 24 17:04:11.992592 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.992555 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:11.995531 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:11.995508 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:12.400998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:12.400920 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:12.402698 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:12.402675 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d8cbf98d5-bg4mh" Jul 24 17:04:12.921729 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:12.921691 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b95cb83_2ecb_47a0_97f7_90b8a8b0b981.slice/crio-2efb549f079d21b9d2161570436635faf24d9bc9556f2920f2c92470d563c45d WatchSource:0}: Error finding container 2efb549f079d21b9d2161570436635faf24d9bc9556f2920f2c92470d563c45d: Status 404 returned error can't find the container with id 2efb549f079d21b9d2161570436635faf24d9bc9556f2920f2c92470d563c45d Jul 24 17:04:12.923896 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:12.923861 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b26b2d_e39a_468f_8bb9_12a99af5881b.slice/crio-f4ffb92b19c6e96b8bd29a3a50b668a2851349929bd8e4a60bf25fd0e064b1ea WatchSource:0}: Error finding container f4ffb92b19c6e96b8bd29a3a50b668a2851349929bd8e4a60bf25fd0e064b1ea: Status 404 returned error can't find the container with id f4ffb92b19c6e96b8bd29a3a50b668a2851349929bd8e4a60bf25fd0e064b1ea Jul 24 17:04:12.928667 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:12.928572 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e2c9e0_2432_4ab2_bfe6_6669caccc4e5.slice/crio-58b21405984fb4fd034d7dfb46997037bd0c8ddbb658092457abe12191e2b418 WatchSource:0}: Error finding container 58b21405984fb4fd034d7dfb46997037bd0c8ddbb658092457abe12191e2b418: Status 404 returned error can't find the container with id 58b21405984fb4fd034d7dfb46997037bd0c8ddbb658092457abe12191e2b418 Jul 24 17:04:13.130967 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.130875 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-gr78g"] Jul 24 17:04:13.282353 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:13.282276 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e023d0_0c62_4c0e_a105_c4c18106d90e.slice/crio-327c5c2dfc5ef4406626464e67f81fa054cda89819a7bbd7dc3dbf73bd253a2e WatchSource:0}: Error finding container 327c5c2dfc5ef4406626464e67f81fa054cda89819a7bbd7dc3dbf73bd253a2e: Status 404 returned error can't find the container with id 327c5c2dfc5ef4406626464e67f81fa054cda89819a7bbd7dc3dbf73bd253a2e Jul 24 17:04:13.406779 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.406734 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-gr78g" event={"ID":"c1e023d0-0c62-4c0e-a105-c4c18106d90e","Type":"ContainerStarted","Data":"327c5c2dfc5ef4406626464e67f81fa054cda89819a7bbd7dc3dbf73bd253a2e"} Jul 24 17:04:13.408244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.408167 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" event={"ID":"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5","Type":"ContainerStarted","Data":"58b21405984fb4fd034d7dfb46997037bd0c8ddbb658092457abe12191e2b418"} Jul 24 17:04:13.409966 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.409941 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" event={"ID":"b1b26b2d-e39a-468f-8bb9-12a99af5881b","Type":"ContainerStarted","Data":"f4ffb92b19c6e96b8bd29a3a50b668a2851349929bd8e4a60bf25fd0e064b1ea"} Jul 24 17:04:13.411485 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.411459 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" event={"ID":"b40c90cb-7c97-488e-83b0-b243919bbec3","Type":"ContainerStarted","Data":"2c1dbdf57d4a08bcd460710a92b614d3f8eba94b4ef3b8ca7997a160ce755737"} Jul 24 17:04:13.413262 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.413236 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8pxz" event={"ID":"895d8902-9011-45c2-8273-5beb236a0a5a","Type":"ContainerStarted","Data":"4aa2e1a5e78a5b32ba8c4f4739802cd11ab47e8884631c247d2d7b29b51adee4"} Jul 24 17:04:13.414609 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.414583 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" event={"ID":"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981","Type":"ContainerStarted","Data":"2efb549f079d21b9d2161570436635faf24d9bc9556f2920f2c92470d563c45d"} Jul 24 17:04:13.868829 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.868794 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:13.925911 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:13.925851 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:14.686576 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.685115 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-578f445665-wlr4k"] Jul 24 17:04:14.727733 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.727700 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-578f445665-wlr4k"] Jul 24 17:04:14.727916 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.727829 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:14.731543 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.731195 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 17:04:14.732489 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.732100 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b99bk\"" Jul 24 17:04:14.732489 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.732316 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 17:04:14.733225 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.732759 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 17:04:14.733225 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.732968 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 17:04:14.733225 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.733168 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 17:04:14.901814 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.901774 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-console-config\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:14.902003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.901833 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-oauth-config\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:14.902003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.901874 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-service-ca\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:14.902003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.901914 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-oauth-serving-cert\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:14.902003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.901954 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-serving-cert\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:14.902003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:14.901994 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxhb\" (UniqueName: \"kubernetes.io/projected/997804c8-51d8-453b-9225-e1f8e0331399-kube-api-access-cpxhb\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.002786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.002704 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-oauth-serving-cert\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.002786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.002772 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-serving-cert\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.003021 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.002809 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxhb\" (UniqueName: \"kubernetes.io/projected/997804c8-51d8-453b-9225-e1f8e0331399-kube-api-access-cpxhb\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.003021 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.002857 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-console-config\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.003021 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.002895 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-oauth-config\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.003193 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.003024 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-service-ca\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.003981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.003950 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-service-ca\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.003981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.003973 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-console-config\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.005119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.005096 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-oauth-serving-cert\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.008907 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.008867 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-serving-cert\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.010160 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.010137 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-oauth-config\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.016001 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.015960 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxhb\" (UniqueName: \"kubernetes.io/projected/997804c8-51d8-453b-9225-e1f8e0331399-kube-api-access-cpxhb\") pod \"console-578f445665-wlr4k\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.042241 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.042037 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:15.281758 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.281612 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:15.382697 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.382655 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:15.586846 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.586732 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:15.688724 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:15.688682 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:16.206686 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:16.206650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:16.307896 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:16.307856 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:16.666623 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:16.666585 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:16.762616 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:16.762509 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:16.818519 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:16.818483 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:17.280682 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:17.280578 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:17.338888 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:17.338843 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:19.536701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:19.536375 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-578f445665-wlr4k"] Jul 24 17:04:20.045460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.045206 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:20.128008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.127974 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:20.463127 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.463090 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg95g" event={"ID":"86216dce-69aa-4dc1-9231-85ad8505db50","Type":"ContainerStarted","Data":"57a9036e9e979abc57495074da0198357875f5b27580cbdc2a5071055aec5349"} Jul 24 17:04:20.465765 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.465723 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" event={"ID":"0e0f63bd-220f-46f4-b52c-60dccf2f6e2f","Type":"ContainerStarted","Data":"6b27ab8868683365386c0cacaf97016f8f3f48dc76ae206e1df737c94911edbe"} Jul 24 17:04:20.469292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.468822 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8pxz" event={"ID":"895d8902-9011-45c2-8273-5beb236a0a5a","Type":"ContainerStarted","Data":"d1518fc182a6c56e282c823dfdbbc513af7fb8a36ae98b4b9fc282b800aa2302"} Jul 24 17:04:20.471485 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.471461 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" event={"ID":"4b95cb83-2ecb-47a0-97f7-90b8a8b0b981","Type":"ContainerStarted","Data":"855778174c2c6457f8d2eddb4545725199ac6c3f52d643ee98a199305d53f95c"} Jul 24 17:04:20.473837 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.473781 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-578f445665-wlr4k" event={"ID":"997804c8-51d8-453b-9225-e1f8e0331399","Type":"ContainerStarted","Data":"dc5d9d6fd1ebb6bb0748d686f7548d09b5efbe5bab013f3f06bb018d7a54becc"} Jul 24 17:04:20.476982 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.476951 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sq9g2" event={"ID":"45bb5108-6076-46d7-9421-ca8ea674fe31","Type":"ContainerStarted","Data":"bebcb768951fc616e3e6dae465449ec11d4b6e424a2a3a8c44cd4f3b51de7e93"} Jul 24 17:04:20.477083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.477003 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sq9g2" event={"ID":"45bb5108-6076-46d7-9421-ca8ea674fe31","Type":"ContainerStarted","Data":"985f1dd23032826d7edca0302a73000235c509e603c050ac96e6f2ff84ff0fd0"} Jul 24 17:04:20.477224 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.477202 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sq9g2" Jul 24 17:04:20.480017 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.479990 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmnnx" event={"ID":"b4985b5b-1bdb-4bd8-b250-67e63f89d01f","Type":"ContainerStarted","Data":"ee4eea416b2ba6907c0fe871485f93da84eb5751be35e3f1de1f807e74fea664"} Jul 24 17:04:20.507315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.507112 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vf6rz" podStartSLOduration=34.549155311 podStartE2EDuration="43.507095556s" totalImagesPullingTime="8.957940245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:20.490353154 +0000 UTC m=+77.988755701" watchObservedRunningTime="2026-07-24 17:04:20.507095556 +0000 UTC m=+78.005498184" Jul 24 17:04:20.507940 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.507783 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w8pxz" podStartSLOduration=74.796977964 podStartE2EDuration="1m17.507771208s" totalImagesPullingTime="2.710793244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:20.506740075 +0000 UTC m=+78.005142622" watchObservedRunningTime="2026-07-24 17:04:20.507771208 +0000 UTC m=+78.006173755" Jul 24 17:04:20.529890 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.529479 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-zsp7b" podStartSLOduration=3.111189211 podStartE2EDuration="9.52946115s" totalImagesPullingTime="6.418271939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:20.528936708 +0000 UTC m=+78.027339255" watchObservedRunningTime="2026-07-24 17:04:20.52946115 +0000 UTC m=+78.027863674" Jul 24 17:04:20.548999 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.548391 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sq9g2" podStartSLOduration=37.91292298 podStartE2EDuration="43.548369833s" totalImagesPullingTime="5.635446853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:20.547792289 +0000 UTC m=+78.046194838" watchObservedRunningTime="2026-07-24 17:04:20.548369833 +0000 UTC m=+78.046772380" Jul 24 17:04:20.563981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.563702 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zmnnx" podStartSLOduration=68.435752917 podStartE2EDuration="1m17.563682362s" totalImagesPullingTime="9.127929445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:20.562035427 +0000 UTC m=+78.060437965" watchObservedRunningTime="2026-07-24 17:04:20.563682362 +0000 UTC m=+78.062084910" Jul 24 17:04:20.566615 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.566450 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:20.667172 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.667134 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:20.768961 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.768877 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:20.872222 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.872173 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:20.949832 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:20.949771 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:21.463926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:21.463879 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:21.520202 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:21.520154 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:23.498338 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:23.498273 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" event={"ID":"c0e2c9e0-2432-4ab2-bfe6-6669caccc4e5","Type":"ContainerStarted","Data":"6a231e3d869462e82b4fa77f2b80de7c606013ce39d1eafcd41c44de7f9478b7"} Jul 24 17:04:23.500496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:23.500451 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" event={"ID":"b1b26b2d-e39a-468f-8bb9-12a99af5881b","Type":"ContainerStarted","Data":"e7a5bbc091935367a6b04802fcbe2469cc85ee3827a423fadf70846220f36fc7"} Jul 24 17:04:23.500741 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:23.500681 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:23.502209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:23.502180 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" Jul 24 17:04:23.512873 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:23.512824 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b4d55ffd-jnbwm" podStartSLOduration=2.197405286 podStartE2EDuration="12.512807052s" totalImagesPullingTime="10.315401766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:23.511822974 +0000 UTC m=+81.010225521" watchObservedRunningTime="2026-07-24 17:04:23.512807052 +0000 UTC m=+81.011209599" Jul 24 17:04:23.532627 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:23.532531 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddd47b78d-gtfvc" podStartSLOduration=2.198794819 podStartE2EDuration="12.532514635s" totalImagesPullingTime="10.333719816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:23.529543539 +0000 UTC m=+81.027946084" watchObservedRunningTime="2026-07-24 17:04:23.532514635 +0000 UTC m=+81.030917184" Jul 24 17:04:24.727859 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:24.727825 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:24.786423 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:24.786365 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:29.095792 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:29.095745 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:29.159489 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:29.159450 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:29.986663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:29.986629 2552 patch_prober.go:28] interesting pod/image-registry-6566bdd846-59cjr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 17:04:29.986906 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:29.986705 2552 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:04:30.151195 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:30.151157 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:30.209397 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:30.209306 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:31.403179 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:31.403040 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:31.458713 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:31.458682 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:31.494709 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:31.494661 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sq9g2" Jul 24 17:04:33.352071 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:33.352010 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:33.419891 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:33.419855 2552 patch_prober.go:28] interesting pod/image-registry-6566bdd846-59cjr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 17:04:33.420077 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:33.419912 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:04:34.553946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.553840 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-578f445665-wlr4k" event={"ID":"997804c8-51d8-453b-9225-e1f8e0331399","Type":"ContainerStarted","Data":"a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e"} Jul 24 17:04:34.555750 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.555721 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-gr78g" event={"ID":"c1e023d0-0c62-4c0e-a105-c4c18106d90e","Type":"ContainerStarted","Data":"24f2240333f149ac9533eca80d167683b8c19ed884e41de7ed561d8d60e42c44"} Jul 24 17:04:34.555887 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.555772 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-gr78g" Jul 24 17:04:34.557622 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.557595 2552 patch_prober.go:28] interesting pod/downloads-7c97dc4555-gr78g container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body= Jul 24 17:04:34.557730 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.557646 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-gr78g" podUID="c1e023d0-0c62-4c0e-a105-c4c18106d90e" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" Jul 24 17:04:34.558495 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.558465 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dg95g" event={"ID":"86216dce-69aa-4dc1-9231-85ad8505db50","Type":"ContainerStarted","Data":"4c3ccff44484d711cdccb8f8c865329d4c501134114c1e7ab4f48ce8db6f0c83"} Jul 24 17:04:34.570983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.570936 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-578f445665-wlr4k" podStartSLOduration=5.937040703 podStartE2EDuration="20.570916583s" totalImagesPullingTime="14.63387588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:34.569016406 +0000 UTC m=+92.067418952" watchObservedRunningTime="2026-07-24 17:04:34.570916583 +0000 UTC m=+92.069319134" Jul 24 17:04:34.585238 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.585201 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-gr78g" podStartSLOduration=2.615451009 podStartE2EDuration="23.585187787s" totalImagesPullingTime="20.969736778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:34.58316799 +0000 UTC m=+92.081570536" watchObservedRunningTime="2026-07-24 17:04:34.585187787 +0000 UTC m=+92.083590334" Jul 24 17:04:34.604282 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:34.604237 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dg95g" podStartSLOduration=17.818883656 podStartE2EDuration="46.604220096s" totalImagesPullingTime="28.78533644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:03:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:34.602884778 +0000 UTC m=+92.101287326" watchObservedRunningTime="2026-07-24 17:04:34.604220096 +0000 UTC m=+92.102622646" Jul 24 17:04:35.043483 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:35.043367 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:35.043483 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:35.043449 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:35.049852 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:35.049827 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:35.570279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:35.570245 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:04:35.579194 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:35.579164 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-gr78g" Jul 24 17:04:35.714194 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:35.714161 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.168555 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.168518 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.268483 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.268444 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.368760 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.368717 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.446948 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.445776 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vw9vm"] Jul 24 17:04:36.464708 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.464629 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.469101 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.468348 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.471393 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.471062 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 17:04:36.471393 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.471138 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x7dbk\"" Jul 24 17:04:36.471393 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.471278 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 17:04:36.472850 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.472555 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 17:04:36.472850 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.472560 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501052 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-textfile\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501101 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501133 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-sys\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501172 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-wtmp\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501203 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-metrics-client-ca\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501225 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501256 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-root\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501287 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-tls\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.501476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.501323 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5q5\" (UniqueName: \"kubernetes.io/projected/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-kube-api-access-bx5q5\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.556429 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.556321 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602146 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-textfile\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602213 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602251 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-sys\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602307 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-wtmp\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602339 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-metrics-client-ca\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602365 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602457 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-root\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602481 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-tls\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.602527 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5q5\" (UniqueName: \"kubernetes.io/projected/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-kube-api-access-bx5q5\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.603875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.603188 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-textfile\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.605504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.605388 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-sys\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.608777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.607683 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-wtmp\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.608777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.607771 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-root\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.608777 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:04:36.607858 2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 17:04:36.608777 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:04:36.607916 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-tls podName:58cf2dfc-d07e-4e2e-8bcb-e053a18af71b nodeName:}" failed. No retries permitted until 2026-07-24 17:04:37.107897028 +0000 UTC m=+94.606299555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-tls") pod "node-exporter-vw9vm" (UID: "58cf2dfc-d07e-4e2e-8bcb-e053a18af71b") : secret "node-exporter-tls" not found Jul 24 17:04:36.610099 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.610063 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-metrics-client-ca\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.610338 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.610320 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.620059 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.620015 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.630663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.622888 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5q5\" (UniqueName: \"kubernetes.io/projected/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-kube-api-access-bx5q5\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:36.653284 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.653191 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.755883 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.755722 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.855839 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.855798 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:36.955225 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:36.955154 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:37.053960 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:37.053820 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:37.109075 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:37.109038 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-tls\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:37.113014 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:37.112984 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58cf2dfc-d07e-4e2e-8bcb-e053a18af71b-node-exporter-tls\") pod \"node-exporter-vw9vm\" (UID: \"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b\") " pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:37.380115 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:37.379630 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:37.391811 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:37.391644 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vw9vm" Jul 24 17:04:37.490027 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:37.489920 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:37.855102 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:04:37.855061 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58cf2dfc_d07e_4e2e_8bcb_e053a18af71b.slice/crio-f8f27d5122c4674c7022bc35e4dc673e77caae21124ea273d767cb2b51314054 WatchSource:0}: Error finding container f8f27d5122c4674c7022bc35e4dc673e77caae21124ea273d767cb2b51314054: Status 404 returned error can't find the container with id f8f27d5122c4674c7022bc35e4dc673e77caae21124ea273d767cb2b51314054 Jul 24 17:04:37.990185 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:37.990148 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:38.044028 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:38.043992 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:38.586359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:38.586128 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vw9vm" event={"ID":"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b","Type":"ContainerStarted","Data":"f8f27d5122c4674c7022bc35e4dc673e77caae21124ea273d767cb2b51314054"} Jul 24 17:04:38.587944 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:38.587909 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" event={"ID":"b40c90cb-7c97-488e-83b0-b243919bbec3","Type":"ContainerStarted","Data":"60fed3988e63c7a48018bcae94c9c28f596d12f395ad013febac7c4c2c0d2f01"} Jul 24 17:04:39.005479 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.005446 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.108436 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.108350 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.206458 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.206421 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.340489 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.340395 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.414699 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.414664 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.595757 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.595673 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vw9vm" event={"ID":"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b","Type":"ContainerStarted","Data":"3a4a57f7744b3b4e50e4b3d7723ed36ca74f52a0333abe51defaa4985f5a1fba"} Jul 24 17:04:39.599213 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.599164 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" event={"ID":"b40c90cb-7c97-488e-83b0-b243919bbec3","Type":"ContainerStarted","Data":"4c4879e30c66db91cc6acba75252c1452cb32c3479e62c99c1f0947f88a5085d"} Jul 24 17:04:39.599367 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.599226 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" event={"ID":"b40c90cb-7c97-488e-83b0-b243919bbec3","Type":"ContainerStarted","Data":"7be2049e67bd3f0004ec6fdf23cb13a26cf2101e6bb1a90ff5249f13be2e037d"} Jul 24 17:04:39.623269 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.623234 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.633137 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.633086 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6dccd775-4p7pn" podStartSLOduration=3.207851564 podStartE2EDuration="28.633065321s" totalImagesPullingTime="25.425213757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:39.631317906 +0000 UTC m=+97.129720453" watchObservedRunningTime="2026-07-24 17:04:39.633065321 +0000 UTC m=+97.131467868" Jul 24 17:04:39.731186 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.731148 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.864896 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.864811 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:39.986113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.986059 2552 patch_prober.go:28] interesting pod/image-registry-6566bdd846-59cjr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 17:04:39.986270 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:39.986135 2552 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:04:40.096712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.096652 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6566bdd846-59cjr"] Jul 24 17:04:40.101541 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.101507 2552 patch_prober.go:28] interesting pod/image-registry-6566bdd846-59cjr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 17:04:40.101690 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.101564 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:04:40.187439 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.187309 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:40.286895 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.286860 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:40.609839 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.609759 2552 generic.go:356] "Generic (PLEG): container finished" podID="58cf2dfc-d07e-4e2e-8bcb-e053a18af71b" containerID="3a4a57f7744b3b4e50e4b3d7723ed36ca74f52a0333abe51defaa4985f5a1fba" exitCode=0 Jul 24 17:04:40.610106 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.609850 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vw9vm" event={"ID":"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b","Type":"ContainerDied","Data":"3a4a57f7744b3b4e50e4b3d7723ed36ca74f52a0333abe51defaa4985f5a1fba"} Jul 24 17:04:40.652078 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.652033 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:40.788534 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.788495 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:40.918476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:40.918444 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-578f445665-wlr4k"] Jul 24 17:04:41.026087 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.026049 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:41.148258 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.148168 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:41.245334 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.245273 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:41.337968 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.337909 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:41.400237 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.400153 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-586fh" Jul 24 17:04:41.421653 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.421620 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:41.618127 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.618093 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vw9vm" event={"ID":"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b","Type":"ContainerStarted","Data":"dba4f5a9d30d3dc7b9c416212d6bfd6bfd3cd527da36f47889835a8f2fbbd297"} Jul 24 17:04:41.618335 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.618154 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vw9vm" event={"ID":"58cf2dfc-d07e-4e2e-8bcb-e053a18af71b","Type":"ContainerStarted","Data":"55f320db0c7755d6f727c5158a4dd3cc049ba500affee4036f37f286a51d8c9f"} Jul 24 17:04:41.638028 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:41.637940 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vw9vm" podStartSLOduration=3.945851809 podStartE2EDuration="5.637920044s" totalImagesPullingTime="1.581623002s" totalInitContainerRuntime="110.445233ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:04:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:04:41.636508257 +0000 UTC m=+99.134910802" watchObservedRunningTime="2026-07-24 17:04:41.637920044 +0000 UTC m=+99.136322591" Jul 24 17:04:42.111787 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:42.111740 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:42.180149 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:42.180115 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:43.071392 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:43.071353 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:43.171850 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:43.171809 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:46.209829 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:46.209793 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:46.313684 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:46.313651 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:48.735841 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:48.735798 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:48.836313 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:48.836263 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:48.901726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:48.901677 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:49.662714 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:49.662677 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:04:50.101064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:50.100993 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:04:58.691840 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:58.691805 2552 generic.go:356] "Generic (PLEG): container finished" podID="3265b58a-6a78-4afa-ba20-533d76387071" containerID="e54771ef37ed274f7d3780439498c278de20740d67a9f033ad4fb468d8f169f2" exitCode=0 Jul 24 17:04:58.692279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:58.691881 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-455nt" event={"ID":"3265b58a-6a78-4afa-ba20-533d76387071","Type":"ContainerDied","Data":"e54771ef37ed274f7d3780439498c278de20740d67a9f033ad4fb468d8f169f2"} Jul 24 17:04:58.692330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:58.692279 2552 scope.go:117] "RemoveContainer" containerID="e54771ef37ed274f7d3780439498c278de20740d67a9f033ad4fb468d8f169f2" Jul 24 17:04:58.693486 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:58.693469 2552 generic.go:356] "Generic (PLEG): container finished" podID="bfa92a22-0110-4c21-8ec9-3de926b4cba3" containerID="0c1dc1242024641790d2deb67719f8e894b77f8785bc42997e03f05d270479b6" exitCode=0 Jul 24 17:04:58.693558 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:58.693499 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" event={"ID":"bfa92a22-0110-4c21-8ec9-3de926b4cba3","Type":"ContainerDied","Data":"0c1dc1242024641790d2deb67719f8e894b77f8785bc42997e03f05d270479b6"} Jul 24 17:04:58.693747 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:58.693736 2552 scope.go:117] "RemoveContainer" containerID="0c1dc1242024641790d2deb67719f8e894b77f8785bc42997e03f05d270479b6" Jul 24 17:04:59.699334 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:59.699296 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-xk9wg" event={"ID":"bfa92a22-0110-4c21-8ec9-3de926b4cba3","Type":"ContainerStarted","Data":"6e3433e981935bdbb2624925ac28f48437b0775be1a394f17360e280f6fa3a0b"} Jul 24 17:04:59.701580 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:59.701555 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-455nt" event={"ID":"3265b58a-6a78-4afa-ba20-533d76387071","Type":"ContainerStarted","Data":"51c66bbb6686c5b26c71643f9ec5112fc35c68f52363d8bb525f5196ae0dc594"} Jul 24 17:04:59.845585 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:04:59.845546 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:00.046807 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:00.046731 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:01.208830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:01.208796 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:01.263728 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:01.263697 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:05.118866 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.118827 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" containerID="cri-o://0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6" gracePeriod=30 Jul 24 17:05:05.363079 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.363054 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:05:05.466171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466139 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466184 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-bound-sa-token\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466390 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466214 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-trusted-ca\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466390 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466253 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-installation-pull-secrets\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466390 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466317 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4e20cf2-9271-4dfd-9143-0101267953ab-ca-trust-extracted\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466390 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466356 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d46f7\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-kube-api-access-d46f7\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466403 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-image-registry-private-configuration\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466503 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-certificates\") pod \"c4e20cf2-9271-4dfd-9143-0101267953ab\" (UID: \"c4e20cf2-9271-4dfd-9143-0101267953ab\") " Jul 24 17:05:05.466782 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.466750 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:05:05.467082 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.467053 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:05:05.468782 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.468757 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:05:05.468883 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.468774 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:05:05.469042 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.469020 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:05:05.469137 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.469018 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-kube-api-access-d46f7" (OuterVolumeSpecName: "kube-api-access-d46f7") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "kube-api-access-d46f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:05:05.469311 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.469279 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:05:05.475534 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.475506 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e20cf2-9271-4dfd-9143-0101267953ab-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c4e20cf2-9271-4dfd-9143-0101267953ab" (UID: "c4e20cf2-9271-4dfd-9143-0101267953ab"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:05:05.567377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567347 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d46f7\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-kube-api-access-d46f7\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.567377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567374 2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-image-registry-private-configuration\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.567377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567384 2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-certificates\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.567377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567393 2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-registry-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.567643 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567402 2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c4e20cf2-9271-4dfd-9143-0101267953ab-bound-sa-token\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.567643 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567426 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c4e20cf2-9271-4dfd-9143-0101267953ab-trusted-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.567643 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567435 2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c4e20cf2-9271-4dfd-9143-0101267953ab-installation-pull-secrets\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.567643 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.567442 2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c4e20cf2-9271-4dfd-9143-0101267953ab-ca-trust-extracted\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:05.725852 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.725755 2552 generic.go:356] "Generic (PLEG): container finished" podID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerID="0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6" exitCode=0 Jul 24 17:05:05.725852 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.725795 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" Jul 24 17:05:05.725852 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.725805 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" event={"ID":"c4e20cf2-9271-4dfd-9143-0101267953ab","Type":"ContainerDied","Data":"0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6"} Jul 24 17:05:05.726112 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.725858 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6566bdd846-59cjr" event={"ID":"c4e20cf2-9271-4dfd-9143-0101267953ab","Type":"ContainerDied","Data":"702d2f7abc802fc3fa71debdc39cacacca2ad257614266e94a3773c2224689ba"} Jul 24 17:05:05.726112 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.725884 2552 scope.go:117] "RemoveContainer" containerID="0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6" Jul 24 17:05:05.734522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.734495 2552 scope.go:117] "RemoveContainer" containerID="0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6" Jul 24 17:05:05.734796 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:05:05.734775 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6\": container with ID starting with 0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6 not found: ID does not exist" containerID="0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6" Jul 24 17:05:05.734872 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.734807 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6"} err="failed to get container status \"0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6\": rpc error: code = NotFound desc = could not find container \"0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6\": container with ID starting with 0366fb5e1e8bc69504bb8b2cd75a248317ddc5f543f802dbd443f8c831ab55a6 not found: ID does not exist" Jul 24 17:05:05.747630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.747610 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6566bdd846-59cjr"] Jul 24 17:05:05.752797 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.752776 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6566bdd846-59cjr"] Jul 24 17:05:05.942067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:05.942015 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-578f445665-wlr4k" podUID="997804c8-51d8-453b-9225-e1f8e0331399" containerName="console" containerID="cri-o://a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e" gracePeriod=15 Jul 24 17:05:06.189359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.189333 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:05:06.373251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373168 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-service-ca\") pod \"997804c8-51d8-453b-9225-e1f8e0331399\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " Jul 24 17:05:06.373251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373209 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-oauth-config\") pod \"997804c8-51d8-453b-9225-e1f8e0331399\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " Jul 24 17:05:06.373251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373240 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-serving-cert\") pod \"997804c8-51d8-453b-9225-e1f8e0331399\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " Jul 24 17:05:06.373562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373382 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpxhb\" (UniqueName: \"kubernetes.io/projected/997804c8-51d8-453b-9225-e1f8e0331399-kube-api-access-cpxhb\") pod \"997804c8-51d8-453b-9225-e1f8e0331399\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " Jul 24 17:05:06.373562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373455 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-console-config\") pod \"997804c8-51d8-453b-9225-e1f8e0331399\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " Jul 24 17:05:06.373562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373494 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-oauth-serving-cert\") pod \"997804c8-51d8-453b-9225-e1f8e0331399\" (UID: \"997804c8-51d8-453b-9225-e1f8e0331399\") " Jul 24 17:05:06.373827 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373719 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-service-ca" (OuterVolumeSpecName: "service-ca") pod "997804c8-51d8-453b-9225-e1f8e0331399" (UID: "997804c8-51d8-453b-9225-e1f8e0331399"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:05:06.373908 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373842 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-console-config" (OuterVolumeSpecName: "console-config") pod "997804c8-51d8-453b-9225-e1f8e0331399" (UID: "997804c8-51d8-453b-9225-e1f8e0331399"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:05:06.373908 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.373875 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "997804c8-51d8-453b-9225-e1f8e0331399" (UID: "997804c8-51d8-453b-9225-e1f8e0331399"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:05:06.375674 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.375640 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997804c8-51d8-453b-9225-e1f8e0331399-kube-api-access-cpxhb" (OuterVolumeSpecName: "kube-api-access-cpxhb") pod "997804c8-51d8-453b-9225-e1f8e0331399" (UID: "997804c8-51d8-453b-9225-e1f8e0331399"). InnerVolumeSpecName "kube-api-access-cpxhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:05:06.375674 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.375643 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "997804c8-51d8-453b-9225-e1f8e0331399" (UID: "997804c8-51d8-453b-9225-e1f8e0331399"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:05:06.375806 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.375710 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "997804c8-51d8-453b-9225-e1f8e0331399" (UID: "997804c8-51d8-453b-9225-e1f8e0331399"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:05:06.475155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.475121 2552 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:06.475155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.475149 2552 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-oauth-config\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:06.475155 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.475158 2552 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997804c8-51d8-453b-9225-e1f8e0331399-console-serving-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:06.475358 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.475169 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpxhb\" (UniqueName: \"kubernetes.io/projected/997804c8-51d8-453b-9225-e1f8e0331399-kube-api-access-cpxhb\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:06.475358 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.475180 2552 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-console-config\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:06.475358 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.475189 2552 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997804c8-51d8-453b-9225-e1f8e0331399-oauth-serving-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:05:06.731209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.731170 2552 generic.go:356] "Generic (PLEG): container finished" podID="997804c8-51d8-453b-9225-e1f8e0331399" containerID="a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e" exitCode=0 Jul 24 17:05:06.731377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.731220 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-578f445665-wlr4k" Jul 24 17:05:06.731377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.731256 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-578f445665-wlr4k" event={"ID":"997804c8-51d8-453b-9225-e1f8e0331399","Type":"ContainerDied","Data":"a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e"} Jul 24 17:05:06.731377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.731329 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-578f445665-wlr4k" event={"ID":"997804c8-51d8-453b-9225-e1f8e0331399","Type":"ContainerDied","Data":"dc5d9d6fd1ebb6bb0748d686f7548d09b5efbe5bab013f3f06bb018d7a54becc"} Jul 24 17:05:06.731377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.731350 2552 scope.go:117] "RemoveContainer" containerID="a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e" Jul 24 17:05:06.740444 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.740401 2552 scope.go:117] "RemoveContainer" containerID="a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e" Jul 24 17:05:06.740708 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:05:06.740690 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e\": container with ID starting with a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e not found: ID does not exist" containerID="a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e" Jul 24 17:05:06.740764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.740715 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e"} err="failed to get container status \"a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e\": rpc error: code = NotFound desc = could not find container \"a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e\": container with ID starting with a32bd0b5e9b99ede6dd174567b50800f84a6333f802672b5a29ab01c82c5133e not found: ID does not exist" Jul 24 17:05:06.752629 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.752604 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-578f445665-wlr4k"] Jul 24 17:05:06.757508 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.757483 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-578f445665-wlr4k"] Jul 24 17:05:06.963048 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.963015 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="997804c8-51d8-453b-9225-e1f8e0331399" path="/var/lib/kubelet/pods/997804c8-51d8-453b-9225-e1f8e0331399/volumes" Jul 24 17:05:06.963437 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:06.963423 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" path="/var/lib/kubelet/pods/c4e20cf2-9271-4dfd-9143-0101267953ab/volumes" Jul 24 17:05:11.436385 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:11.436346 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:11.537220 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:11.537149 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:12.768743 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:12.768709 2552 generic.go:356] "Generic (PLEG): container finished" podID="477521c4-4d67-4c6e-95b9-44dfb810a695" containerID="ba8d75f4f93f9eff418438634da17306e7ab8cc0b71790d22feed3e8ac12c917" exitCode=0 Jul 24 17:05:12.769110 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:12.768779 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" event={"ID":"477521c4-4d67-4c6e-95b9-44dfb810a695","Type":"ContainerDied","Data":"ba8d75f4f93f9eff418438634da17306e7ab8cc0b71790d22feed3e8ac12c917"} Jul 24 17:05:12.769110 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:12.769102 2552 scope.go:117] "RemoveContainer" containerID="ba8d75f4f93f9eff418438634da17306e7ab8cc0b71790d22feed3e8ac12c917" Jul 24 17:05:13.774308 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:13.774269 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-dnlgj" event={"ID":"477521c4-4d67-4c6e-95b9-44dfb810a695","Type":"ContainerStarted","Data":"fe75f102874cc3a28f804d5c9fffd0e1aa891ac07e94a12f2dd9066c7399292d"} Jul 24 17:05:22.768635 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:22.768585 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:22.818943 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:22.818906 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:52.498899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:52.498860 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:52.554259 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:52.554211 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:59.402565 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:59.402473 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:05:59.460399 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:05:59.460363 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:04.414692 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:04.414650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:04.465139 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:04.465110 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:08.328522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:08.328484 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:10.130496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:10.130458 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:10.232026 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:10.231995 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:10.332927 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:10.332893 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:16.221474 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:16.221433 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:16.318651 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:16.318613 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:28.295886 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:28.295843 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:28.350363 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:28.350332 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:29.958480 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:29.958437 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:43.577818 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:43.577784 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:06:43.671564 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:06:43.671521 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:10.244350 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:10.244287 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:11.024871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:11.024836 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:13.860192 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:13.860155 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:13.962121 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:13.962082 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:15.127985 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:15.127950 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:15.345282 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:15.345242 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:15.501570 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:15.501534 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:17.624532 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:17.624498 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:18.014802 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:18.014763 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:20.652374 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:20.652337 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:20.752735 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:20.752698 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:25.618776 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:25.618687 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:32.059357 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:32.059314 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:32.158266 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:32.158214 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:39.034874 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:39.034838 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:39.140479 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:39.136382 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:41.152012 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:41.151970 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:46.688384 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:46.688345 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:46.745027 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:46.744992 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:51.365788 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:51.365749 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:51.465354 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:51.465324 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:52.861330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:52.861300 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:52.971652 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:52.971618 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:53.511498 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.511464 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:53.838382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.838297 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:53.913650 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.913623 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:53.970509 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.970479 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf"] Jul 24 17:07:53.970765 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.970753 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="997804c8-51d8-453b-9225-e1f8e0331399" containerName="console" Jul 24 17:07:53.970809 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.970767 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="997804c8-51d8-453b-9225-e1f8e0331399" containerName="console" Jul 24 17:07:53.970809 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.970784 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" Jul 24 17:07:53.970809 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.970790 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" Jul 24 17:07:53.970901 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.970837 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e20cf2-9271-4dfd-9143-0101267953ab" containerName="registry" Jul 24 17:07:53.970901 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.970844 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="997804c8-51d8-453b-9225-e1f8e0331399" containerName="console" Jul 24 17:07:53.972919 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.972904 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:53.975259 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.975232 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 17:07:53.975400 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.975315 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 17:07:53.975400 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.975359 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 17:07:53.975629 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.975581 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ltj8k\"" Jul 24 17:07:53.975807 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.975787 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 17:07:53.987214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:53.987192 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf"] Jul 24 17:07:54.110351 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.110265 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:54.119308 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.119286 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqfgc\" (UniqueName: \"kubernetes.io/projected/81f2ef33-dd41-4967-82d7-ccc90d86c242-kube-api-access-pqfgc\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.119378 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.119340 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81f2ef33-dd41-4967-82d7-ccc90d86c242-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.119449 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.119434 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81f2ef33-dd41-4967-82d7-ccc90d86c242-manifests\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.119492 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.119465 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81f2ef33-dd41-4967-82d7-ccc90d86c242-tmp\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.119492 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.119480 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81f2ef33-dd41-4967-82d7-ccc90d86c242-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.216565 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.216533 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:54.220503 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.220480 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81f2ef33-dd41-4967-82d7-ccc90d86c242-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.220595 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.220528 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81f2ef33-dd41-4967-82d7-ccc90d86c242-manifests\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.220595 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.220561 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81f2ef33-dd41-4967-82d7-ccc90d86c242-tmp\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.220595 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.220585 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81f2ef33-dd41-4967-82d7-ccc90d86c242-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.220739 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.220618 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqfgc\" (UniqueName: \"kubernetes.io/projected/81f2ef33-dd41-4967-82d7-ccc90d86c242-kube-api-access-pqfgc\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.220985 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.220959 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81f2ef33-dd41-4967-82d7-ccc90d86c242-manifests\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.222899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.222878 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81f2ef33-dd41-4967-82d7-ccc90d86c242-tmp\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.223047 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.223029 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81f2ef33-dd41-4967-82d7-ccc90d86c242-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.223193 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.223173 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81f2ef33-dd41-4967-82d7-ccc90d86c242-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.234382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.234354 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqfgc\" (UniqueName: \"kubernetes.io/projected/81f2ef33-dd41-4967-82d7-ccc90d86c242-kube-api-access-pqfgc\") pod \"opendatahub-operator-controller-manager-6bc8c4649b-6dlvf\" (UID: \"81f2ef33-dd41-4967-82d7-ccc90d86c242\") " pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.283131 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.283108 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:07:54.310760 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.310731 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:54.402051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.402018 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:54.413884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.413863 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf"] Jul 24 17:07:54.416530 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:07:54.416503 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f2ef33_dd41_4967_82d7_ccc90d86c242.slice/crio-6bebb31d30489869a72711dbf3cb12ff575884f8ee0b4a6f5caa1c3ab68dead2 WatchSource:0}: Error finding container 6bebb31d30489869a72711dbf3cb12ff575884f8ee0b4a6f5caa1c3ab68dead2: Status 404 returned error can't find the container with id 6bebb31d30489869a72711dbf3cb12ff575884f8ee0b4a6f5caa1c3ab68dead2 Jul 24 17:07:54.504388 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.504353 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:54.579528 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.579490 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:54.682162 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:54.682134 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:07:55.393768 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:55.393729 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" event={"ID":"81f2ef33-dd41-4967-82d7-ccc90d86c242","Type":"ContainerStarted","Data":"6bebb31d30489869a72711dbf3cb12ff575884f8ee0b4a6f5caa1c3ab68dead2"} Jul 24 17:07:58.407228 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:58.407192 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" event={"ID":"81f2ef33-dd41-4967-82d7-ccc90d86c242","Type":"ContainerStarted","Data":"013f394e5beea92d1ee5d2b49aafb3d71afef35fd3e7a2ecac42eba7a18a3868"} Jul 24 17:07:59.412405 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:59.412371 2552 generic.go:356] "Generic (PLEG): container finished" podID="81f2ef33-dd41-4967-82d7-ccc90d86c242" containerID="013f394e5beea92d1ee5d2b49aafb3d71afef35fd3e7a2ecac42eba7a18a3868" exitCode=0 Jul 24 17:07:59.412844 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:07:59.412464 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" event={"ID":"81f2ef33-dd41-4967-82d7-ccc90d86c242","Type":"ContainerDied","Data":"013f394e5beea92d1ee5d2b49aafb3d71afef35fd3e7a2ecac42eba7a18a3868"} Jul 24 17:08:00.419086 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:00.419055 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" event={"ID":"81f2ef33-dd41-4967-82d7-ccc90d86c242","Type":"ContainerStarted","Data":"121af93a7f40693f46fe27a7e155fa9c12ea48a9c42d4305ebc8a384542ef53a"} Jul 24 17:08:00.419513 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:00.419125 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:08:00.440845 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:00.440794 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" podStartSLOduration=3.01378577 podStartE2EDuration="7.440778451s" totalImagesPullingTime="3.378617559s" totalInitContainerRuntime="1.048375122s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:07:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:00.439524404 +0000 UTC m=+297.937926955" watchObservedRunningTime="2026-07-24 17:08:00.440778451 +0000 UTC m=+297.939180997" Jul 24 17:08:02.875307 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:02.875281 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:08:02.877830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:02.877803 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:08:02.888787 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:02.888769 2552 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 17:08:11.425950 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:11.425913 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bc8c4649b-6dlvf" Jul 24 17:08:13.065291 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:13.065258 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:14.331956 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:14.331916 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:23.858559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.858516 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:23.961602 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.961573 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw"] Jul 24 17:08:23.964067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.964047 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:23.965712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.965691 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/252e838b-cd9b-4247-915e-4499396af2d5-tmp\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:23.965808 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.965721 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qr46\" (UniqueName: \"kubernetes.io/projected/252e838b-cd9b-4247-915e-4499396af2d5-kube-api-access-8qr46\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:23.965863 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.965828 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e838b-cd9b-4247-915e-4499396af2d5-tls-certs\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:23.966660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.966642 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 17:08:23.966716 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.966656 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 17:08:23.966716 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.966675 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bmcwf\"" Jul 24 17:08:23.977909 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:23.977889 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw"] Jul 24 17:08:24.066176 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.066139 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e838b-cd9b-4247-915e-4499396af2d5-tls-certs\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.066176 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.066180 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/252e838b-cd9b-4247-915e-4499396af2d5-tmp\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.066386 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.066199 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qr46\" (UniqueName: \"kubernetes.io/projected/252e838b-cd9b-4247-915e-4499396af2d5-kube-api-access-8qr46\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.066386 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:08:24.066300 2552 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 24 17:08:24.066386 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:08:24.066385 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252e838b-cd9b-4247-915e-4499396af2d5-tls-certs podName:252e838b-cd9b-4247-915e-4499396af2d5 nodeName:}" failed. No retries permitted until 2026-07-24 17:08:24.566363048 +0000 UTC m=+322.064765584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/252e838b-cd9b-4247-915e-4499396af2d5-tls-certs") pod "kube-auth-proxy-6d6f9cf76-xx6qw" (UID: "252e838b-cd9b-4247-915e-4499396af2d5") : secret "kube-auth-proxy-tls" not found Jul 24 17:08:24.068533 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.068510 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/252e838b-cd9b-4247-915e-4499396af2d5-tmp\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.080996 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.080974 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qr46\" (UniqueName: \"kubernetes.io/projected/252e838b-cd9b-4247-915e-4499396af2d5-kube-api-access-8qr46\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.167104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.167073 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rs69p"] Jul 24 17:08:24.169504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.169484 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.171831 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.171812 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 17:08:24.172077 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.172065 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8mbs7\"" Jul 24 17:08:24.184863 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.184840 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rs69p"] Jul 24 17:08:24.192154 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.192110 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:24.267513 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.267473 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skcz9\" (UniqueName: \"kubernetes.io/projected/39d73d61-c5d6-49e6-99e4-679136058f81-kube-api-access-skcz9\") pod \"odh-model-controller-7998c67dc4-rs69p\" (UID: \"39d73d61-c5d6-49e6-99e4-679136058f81\") " pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.267666 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.267545 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39d73d61-c5d6-49e6-99e4-679136058f81-cert\") pod \"odh-model-controller-7998c67dc4-rs69p\" (UID: \"39d73d61-c5d6-49e6-99e4-679136058f81\") " pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.286862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.286836 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:24.368308 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.368279 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39d73d61-c5d6-49e6-99e4-679136058f81-cert\") pod \"odh-model-controller-7998c67dc4-rs69p\" (UID: \"39d73d61-c5d6-49e6-99e4-679136058f81\") " pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.368472 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.368368 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skcz9\" (UniqueName: \"kubernetes.io/projected/39d73d61-c5d6-49e6-99e4-679136058f81-kube-api-access-skcz9\") pod \"odh-model-controller-7998c67dc4-rs69p\" (UID: \"39d73d61-c5d6-49e6-99e4-679136058f81\") " pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.368472 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:08:24.368425 2552 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 17:08:24.368591 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:08:24.368493 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39d73d61-c5d6-49e6-99e4-679136058f81-cert podName:39d73d61-c5d6-49e6-99e4-679136058f81 nodeName:}" failed. No retries permitted until 2026-07-24 17:08:24.868478594 +0000 UTC m=+322.366881119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/39d73d61-c5d6-49e6-99e4-679136058f81-cert") pod "odh-model-controller-7998c67dc4-rs69p" (UID: "39d73d61-c5d6-49e6-99e4-679136058f81") : secret "odh-model-controller-webhook-cert" not found Jul 24 17:08:24.376812 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.376786 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skcz9\" (UniqueName: \"kubernetes.io/projected/39d73d61-c5d6-49e6-99e4-679136058f81-kube-api-access-skcz9\") pod \"odh-model-controller-7998c67dc4-rs69p\" (UID: \"39d73d61-c5d6-49e6-99e4-679136058f81\") " pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.441031 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.440960 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:24.570633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.570602 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e838b-cd9b-4247-915e-4499396af2d5-tls-certs\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.572991 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.572955 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e838b-cd9b-4247-915e-4499396af2d5-tls-certs\") pod \"kube-auth-proxy-6d6f9cf76-xx6qw\" (UID: \"252e838b-cd9b-4247-915e-4499396af2d5\") " pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.573674 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.573656 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" Jul 24 17:08:24.706099 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.706022 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw"] Jul 24 17:08:24.709545 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:08:24.709519 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252e838b_cd9b_4247_915e_4499396af2d5.slice/crio-f75d01796cbe1878b666ee002fb66fb6053377b0f75b0823f3c093aba0e4abf3 WatchSource:0}: Error finding container f75d01796cbe1878b666ee002fb66fb6053377b0f75b0823f3c093aba0e4abf3: Status 404 returned error can't find the container with id f75d01796cbe1878b666ee002fb66fb6053377b0f75b0823f3c093aba0e4abf3 Jul 24 17:08:24.711303 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.711286 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:08:24.873284 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.873253 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39d73d61-c5d6-49e6-99e4-679136058f81-cert\") pod \"odh-model-controller-7998c67dc4-rs69p\" (UID: \"39d73d61-c5d6-49e6-99e4-679136058f81\") " pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.875578 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.875560 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39d73d61-c5d6-49e6-99e4-679136058f81-cert\") pod \"odh-model-controller-7998c67dc4-rs69p\" (UID: \"39d73d61-c5d6-49e6-99e4-679136058f81\") " pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:24.948790 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.948756 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:24.992868 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.992790 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9wwg9"] Jul 24 17:08:24.996301 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.996285 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:24.998782 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.998760 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 17:08:24.998934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:24.998915 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2zm5d\"" Jul 24 17:08:25.007763 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.007742 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9wwg9"] Jul 24 17:08:25.075740 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.075713 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba587b9-b59e-4278-9df4-bc874d9414a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-9wwg9\" (UID: \"cba587b9-b59e-4278-9df4-bc874d9414a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:25.075899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.075754 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jgk\" (UniqueName: \"kubernetes.io/projected/cba587b9-b59e-4278-9df4-bc874d9414a1-kube-api-access-85jgk\") pod \"model-serving-api-54d89bd5fc-9wwg9\" (UID: \"cba587b9-b59e-4278-9df4-bc874d9414a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:25.079671 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.079650 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:25.176666 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.176631 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba587b9-b59e-4278-9df4-bc874d9414a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-9wwg9\" (UID: \"cba587b9-b59e-4278-9df4-bc874d9414a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:25.176838 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.176686 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jgk\" (UniqueName: \"kubernetes.io/projected/cba587b9-b59e-4278-9df4-bc874d9414a1-kube-api-access-85jgk\") pod \"model-serving-api-54d89bd5fc-9wwg9\" (UID: \"cba587b9-b59e-4278-9df4-bc874d9414a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:25.176838 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:08:25.176819 2552 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 24 17:08:25.176962 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:08:25.176905 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cba587b9-b59e-4278-9df4-bc874d9414a1-tls-certs podName:cba587b9-b59e-4278-9df4-bc874d9414a1 nodeName:}" failed. No retries permitted until 2026-07-24 17:08:25.676884271 +0000 UTC m=+323.175286797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cba587b9-b59e-4278-9df4-bc874d9414a1-tls-certs") pod "model-serving-api-54d89bd5fc-9wwg9" (UID: "cba587b9-b59e-4278-9df4-bc874d9414a1") : secret "model-serving-api-tls" not found Jul 24 17:08:25.196705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.196661 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jgk\" (UniqueName: \"kubernetes.io/projected/cba587b9-b59e-4278-9df4-bc874d9414a1-kube-api-access-85jgk\") pod \"model-serving-api-54d89bd5fc-9wwg9\" (UID: \"cba587b9-b59e-4278-9df4-bc874d9414a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:25.200423 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.200144 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:25.204209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.203974 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br"] Jul 24 17:08:25.208138 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.208118 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.211444 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.210564 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 17:08:25.211444 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.210814 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 17:08:25.211444 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.211218 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 17:08:25.212184 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.211398 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 17:08:25.212184 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.211737 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-98sdk\"" Jul 24 17:08:25.229074 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.228933 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br"] Jul 24 17:08:25.249170 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.249142 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rs69p"] Jul 24 17:08:25.251992 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:08:25.251963 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d73d61_c5d6_49e6_99e4_679136058f81.slice/crio-5e482ecf7af91e4a0d3ca8d319241a3575b7eac9ecd75d0c65238986c4267d81 WatchSource:0}: Error finding container 5e482ecf7af91e4a0d3ca8d319241a3575b7eac9ecd75d0c65238986c4267d81: Status 404 returned error can't find the container with id 5e482ecf7af91e4a0d3ca8d319241a3575b7eac9ecd75d0c65238986c4267d81 Jul 24 17:08:25.277401 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.277378 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79083a5-ee09-4586-ba1b-eb05e1e560dd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.277550 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.277429 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.277550 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.277486 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.277651 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.277613 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.277697 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.277667 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.277733 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.277696 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwtwq\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-kube-api-access-bwtwq\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.277784 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.277736 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.296824 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.296797 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:25.378802 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.378770 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.378966 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.378811 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwtwq\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-kube-api-access-bwtwq\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.378966 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.378945 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.379081 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.379019 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79083a5-ee09-4586-ba1b-eb05e1e560dd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.379081 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.379048 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.379190 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.379087 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.379190 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.379164 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.379667 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.379623 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.381815 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.381796 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.381936 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.381847 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.381936 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.381847 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.381936 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.381846 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79083a5-ee09-4586-ba1b-eb05e1e560dd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.387924 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.387896 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwtwq\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-kube-api-access-bwtwq\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.388910 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.388891 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-df8br\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.403771 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.403714 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:25.525108 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.525022 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:25.525614 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.525589 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" event={"ID":"252e838b-cd9b-4247-915e-4499396af2d5","Type":"ContainerStarted","Data":"f75d01796cbe1878b666ee002fb66fb6053377b0f75b0823f3c093aba0e4abf3"} Jul 24 17:08:25.526894 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.526864 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" event={"ID":"39d73d61-c5d6-49e6-99e4-679136058f81","Type":"ContainerStarted","Data":"5e482ecf7af91e4a0d3ca8d319241a3575b7eac9ecd75d0c65238986c4267d81"} Jul 24 17:08:25.666683 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.666205 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br"] Jul 24 17:08:25.669044 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:08:25.669008 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79083a5_ee09_4586_ba1b_eb05e1e560dd.slice/crio-5abf81b37d3406e0226e3fea00f4bbf9932d27861c5d81c20fa511e0ee85686e WatchSource:0}: Error finding container 5abf81b37d3406e0226e3fea00f4bbf9932d27861c5d81c20fa511e0ee85686e: Status 404 returned error can't find the container with id 5abf81b37d3406e0226e3fea00f4bbf9932d27861c5d81c20fa511e0ee85686e Jul 24 17:08:25.681952 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.681930 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba587b9-b59e-4278-9df4-bc874d9414a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-9wwg9\" (UID: \"cba587b9-b59e-4278-9df4-bc874d9414a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:25.684651 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.684633 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba587b9-b59e-4278-9df4-bc874d9414a1-tls-certs\") pod \"model-serving-api-54d89bd5fc-9wwg9\" (UID: \"cba587b9-b59e-4278-9df4-bc874d9414a1\") " pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:25.907596 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:25.907565 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:26.085232 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:26.085157 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9wwg9"] Jul 24 17:08:26.087821 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:08:26.087790 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba587b9_b59e_4278_9df4_bc874d9414a1.slice/crio-ab74f84d7e781b0d853d4939e9e6d6f64641b96a771990843bbeaba1159cca90 WatchSource:0}: Error finding container ab74f84d7e781b0d853d4939e9e6d6f64641b96a771990843bbeaba1159cca90: Status 404 returned error can't find the container with id ab74f84d7e781b0d853d4939e9e6d6f64641b96a771990843bbeaba1159cca90 Jul 24 17:08:26.273907 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:26.273781 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:26.535472 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:26.535376 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" event={"ID":"f79083a5-ee09-4586-ba1b-eb05e1e560dd","Type":"ContainerStarted","Data":"5abf81b37d3406e0226e3fea00f4bbf9932d27861c5d81c20fa511e0ee85686e"} Jul 24 17:08:26.536884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:26.536848 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" event={"ID":"cba587b9-b59e-4278-9df4-bc874d9414a1","Type":"ContainerStarted","Data":"ab74f84d7e781b0d853d4939e9e6d6f64641b96a771990843bbeaba1159cca90"} Jul 24 17:08:29.393278 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:29.393230 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:29.493775 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:29.493742 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:30.269843 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.269809 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:08:30.269938 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.269871 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:08:30.560818 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.560729 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" event={"ID":"252e838b-cd9b-4247-915e-4499396af2d5","Type":"ContainerStarted","Data":"0e91b3c4921bb11e73c5dd6b36635349c418ba9a7491c220a05be1641b42b77e"} Jul 24 17:08:30.562347 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.562319 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" event={"ID":"cba587b9-b59e-4278-9df4-bc874d9414a1","Type":"ContainerStarted","Data":"985cc430e09a814b9fd98305a1afae515b1027e6f20895d1a79d2e553631b911"} Jul 24 17:08:30.562494 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.562402 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:30.563924 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.563903 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" event={"ID":"f79083a5-ee09-4586-ba1b-eb05e1e560dd","Type":"ContainerStarted","Data":"76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1"} Jul 24 17:08:30.564028 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.563960 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:30.565334 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.565313 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" event={"ID":"39d73d61-c5d6-49e6-99e4-679136058f81","Type":"ContainerStarted","Data":"53e455d06b69209379a48b36d914489132377ddb44d1668ee133d5f287fd2d75"} Jul 24 17:08:30.565441 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.565367 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:08:30.577793 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.577752 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d6f9cf76-xx6qw" podStartSLOduration=2.026178347 podStartE2EDuration="7.577739383s" totalImagesPullingTime="5.551561036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:30.576799774 +0000 UTC m=+328.075202321" watchObservedRunningTime="2026-07-24 17:08:30.577739383 +0000 UTC m=+328.076141927" Jul 24 17:08:30.594331 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.594265 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" podStartSLOduration=1.571344552 podStartE2EDuration="6.594243946s" totalImagesPullingTime="5.022899394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:30.592986221 +0000 UTC m=+328.091388764" watchObservedRunningTime="2026-07-24 17:08:30.594243946 +0000 UTC m=+328.092646700" Jul 24 17:08:30.613076 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.612347 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" podStartSLOduration=2.435213051 podStartE2EDuration="6.61232663s" totalImagesPullingTime="4.177113579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:30.611591154 +0000 UTC m=+328.109993701" watchObservedRunningTime="2026-07-24 17:08:30.61232663 +0000 UTC m=+328.110729178" Jul 24 17:08:30.636945 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:30.636884 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" podStartSLOduration=1.038461116 podStartE2EDuration="5.636863494s" totalImagesPullingTime="4.598402378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:30.63531398 +0000 UTC m=+328.133716527" watchObservedRunningTime="2026-07-24 17:08:30.636863494 +0000 UTC m=+328.135266042" Jul 24 17:08:31.253673 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:31.253629 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:31.353834 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:31.353786 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:31.574403 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:31.574313 2552 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-df8br container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 17:08:31.574403 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:31.574378 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" podUID="f79083a5-ee09-4586-ba1b-eb05e1e560dd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:08:32.018480 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.018444 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v"] Jul 24 17:08:32.022129 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.022110 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.024894 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.024872 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-gvqqb\"" Jul 24 17:08:32.035546 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.035525 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v"] Jul 24 17:08:32.054192 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.054166 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:32.142136 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142099 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142145 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142207 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c40c231-971d-41de-9487-e66aa463b36b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142263 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142288 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tmj\" (UniqueName: \"kubernetes.io/projected/0c40c231-971d-41de-9487-e66aa463b36b-kube-api-access-g2tmj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142512 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142373 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142512 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142458 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142620 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142507 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c40c231-971d-41de-9487-e66aa463b36b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.142620 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.142541 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c40c231-971d-41de-9487-e66aa463b36b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244024 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.243982 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244036 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tmj\" (UniqueName: \"kubernetes.io/projected/0c40c231-971d-41de-9487-e66aa463b36b-kube-api-access-g2tmj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244107 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244146 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244186 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c40c231-971d-41de-9487-e66aa463b36b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244215 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c40c231-971d-41de-9487-e66aa463b36b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244292 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244330 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244361 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c40c231-971d-41de-9487-e66aa463b36b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244717 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244616 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244717 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244685 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.244985 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.244907 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.245135 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.245108 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c40c231-971d-41de-9487-e66aa463b36b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.245268 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.245178 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.246940 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.246914 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c40c231-971d-41de-9487-e66aa463b36b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.247038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.246945 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c40c231-971d-41de-9487-e66aa463b36b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.257722 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.257697 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tmj\" (UniqueName: \"kubernetes.io/projected/0c40c231-971d-41de-9487-e66aa463b36b-kube-api-access-g2tmj\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.262754 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.262735 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c40c231-971d-41de-9487-e66aa463b36b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v\" (UID: \"0c40c231-971d-41de-9487-e66aa463b36b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.290365 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.290270 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:32.332360 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.332324 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:32.422701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.422667 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:32.492723 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.492695 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v"] Jul 24 17:08:32.495148 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:08:32.495125 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c40c231_971d_41de_9487_e66aa463b36b.slice/crio-dc3551581d80833aa6eaeb3aea8369bd04b0fb8de4ae128cbd165c54bf502b2a WatchSource:0}: Error finding container dc3551581d80833aa6eaeb3aea8369bd04b0fb8de4ae128cbd165c54bf502b2a: Status 404 returned error can't find the container with id dc3551581d80833aa6eaeb3aea8369bd04b0fb8de4ae128cbd165c54bf502b2a Jul 24 17:08:32.576896 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:32.576825 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" event={"ID":"0c40c231-971d-41de-9487-e66aa463b36b","Type":"ContainerStarted","Data":"dc3551581d80833aa6eaeb3aea8369bd04b0fb8de4ae128cbd165c54bf502b2a"} Jul 24 17:08:34.444583 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.444542 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:34.549998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.549967 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx"] Jul 24 17:08:34.553599 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.553510 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.573872 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.573847 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:34.611308 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.611273 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx"] Jul 24 17:08:34.611722 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.611699 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-x46zd\"" Jul 24 17:08:34.611832 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.611741 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 17:08:34.665991 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.665941 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/202119ad-4258-4448-9210-a436ba05064a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rtpmx\" (UID: \"202119ad-4258-4448-9210-a436ba05064a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.665991 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.665985 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2c7c\" (UniqueName: \"kubernetes.io/projected/202119ad-4258-4448-9210-a436ba05064a-kube-api-access-c2c7c\") pod \"llmisvc-controller-manager-777c6bd7d5-rtpmx\" (UID: \"202119ad-4258-4448-9210-a436ba05064a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.726372 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.726289 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:34.767857 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.767823 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/202119ad-4258-4448-9210-a436ba05064a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rtpmx\" (UID: \"202119ad-4258-4448-9210-a436ba05064a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.768036 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.767868 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2c7c\" (UniqueName: \"kubernetes.io/projected/202119ad-4258-4448-9210-a436ba05064a-kube-api-access-c2c7c\") pod \"llmisvc-controller-manager-777c6bd7d5-rtpmx\" (UID: \"202119ad-4258-4448-9210-a436ba05064a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.771063 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.771036 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/202119ad-4258-4448-9210-a436ba05064a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rtpmx\" (UID: \"202119ad-4258-4448-9210-a436ba05064a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.787635 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.787606 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2c7c\" (UniqueName: \"kubernetes.io/projected/202119ad-4258-4448-9210-a436ba05064a-kube-api-access-c2c7c\") pod \"llmisvc-controller-manager-777c6bd7d5-rtpmx\" (UID: \"202119ad-4258-4448-9210-a436ba05064a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.865150 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.865119 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:34.888862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:34.888824 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:35.162630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.162599 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br"] Jul 24 17:08:35.162862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.162821 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" podUID="f79083a5-ee09-4586-ba1b-eb05e1e560dd" containerName="discovery" containerID="cri-o://76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1" gracePeriod=30 Jul 24 17:08:35.708744 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.708666 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:35.793022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.792989 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:08:35.793144 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.793079 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:08:35.793144 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.793133 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:08:35.877521 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.877489 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:35.879057 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.879037 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx"] Jul 24 17:08:35.880544 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:08:35.880523 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod202119ad_4258_4448_9210_a436ba05064a.slice/crio-72f8fbc27086a14be0024620080ce043e54d19aabc367fb7a4cea5fac4b737c0 WatchSource:0}: Error finding container 72f8fbc27086a14be0024620080ce043e54d19aabc367fb7a4cea5fac4b737c0: Status 404 returned error can't find the container with id 72f8fbc27086a14be0024620080ce043e54d19aabc367fb7a4cea5fac4b737c0 Jul 24 17:08:35.896001 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.895980 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:35.980258 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.980167 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-cacerts\") pod \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " Jul 24 17:08:35.980258 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.980205 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-kubeconfig\") pod \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " Jul 24 17:08:35.980522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.980362 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwtwq\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-kube-api-access-bwtwq\") pod \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " Jul 24 17:08:35.980522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.980402 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-dns-cert\") pod \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " Jul 24 17:08:35.980522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.980471 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79083a5-ee09-4586-ba1b-eb05e1e560dd-local-certs\") pod \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " Jul 24 17:08:35.980686 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.980546 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-token\") pod \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " Jul 24 17:08:35.980686 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.980582 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-ca-configmap\") pod \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\" (UID: \"f79083a5-ee09-4586-ba1b-eb05e1e560dd\") " Jul 24 17:08:35.981147 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.981103 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f79083a5-ee09-4586-ba1b-eb05e1e560dd" (UID: "f79083a5-ee09-4586-ba1b-eb05e1e560dd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:08:35.983037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.983003 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "f79083a5-ee09-4586-ba1b-eb05e1e560dd" (UID: "f79083a5-ee09-4586-ba1b-eb05e1e560dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:08:35.983146 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.983064 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f79083a5-ee09-4586-ba1b-eb05e1e560dd" (UID: "f79083a5-ee09-4586-ba1b-eb05e1e560dd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:08:35.983146 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.983077 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-cacerts" (OuterVolumeSpecName: "cacerts") pod "f79083a5-ee09-4586-ba1b-eb05e1e560dd" (UID: "f79083a5-ee09-4586-ba1b-eb05e1e560dd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:08:35.983289 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.983267 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f79083a5-ee09-4586-ba1b-eb05e1e560dd" (UID: "f79083a5-ee09-4586-ba1b-eb05e1e560dd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:08:35.983728 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.983702 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-kube-api-access-bwtwq" (OuterVolumeSpecName: "kube-api-access-bwtwq") pod "f79083a5-ee09-4586-ba1b-eb05e1e560dd" (UID: "f79083a5-ee09-4586-ba1b-eb05e1e560dd"). InnerVolumeSpecName "kube-api-access-bwtwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:08:35.983812 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.983793 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79083a5-ee09-4586-ba1b-eb05e1e560dd-local-certs" (OuterVolumeSpecName: "local-certs") pod "f79083a5-ee09-4586-ba1b-eb05e1e560dd" (UID: "f79083a5-ee09-4586-ba1b-eb05e1e560dd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:08:35.984918 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:35.984890 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:36.082133 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.082102 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwtwq\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-kube-api-access-bwtwq\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:08:36.082133 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.082128 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-dns-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:08:36.082133 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.082137 2552 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79083a5-ee09-4586-ba1b-eb05e1e560dd-local-certs\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:08:36.082431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.082145 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-token\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:08:36.082431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.082153 2552 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-csr-ca-configmap\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:08:36.082431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.082161 2552 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-cacerts\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:08:36.082431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.082170 2552 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f79083a5-ee09-4586-ba1b-eb05e1e560dd-istio-kubeconfig\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:08:36.598842 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.598807 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" event={"ID":"0c40c231-971d-41de-9487-e66aa463b36b","Type":"ContainerStarted","Data":"fa55424bb09c3e4348acfa6a96557bb25e44b33a7f2e1d36878805d8909c0286"} Jul 24 17:08:36.600087 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.600060 2552 generic.go:356] "Generic (PLEG): container finished" podID="f79083a5-ee09-4586-ba1b-eb05e1e560dd" containerID="76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1" exitCode=0 Jul 24 17:08:36.600215 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.600093 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" Jul 24 17:08:36.600215 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.600150 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" event={"ID":"f79083a5-ee09-4586-ba1b-eb05e1e560dd","Type":"ContainerDied","Data":"76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1"} Jul 24 17:08:36.600215 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.600191 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br" event={"ID":"f79083a5-ee09-4586-ba1b-eb05e1e560dd","Type":"ContainerDied","Data":"5abf81b37d3406e0226e3fea00f4bbf9932d27861c5d81c20fa511e0ee85686e"} Jul 24 17:08:36.600215 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.600215 2552 scope.go:117] "RemoveContainer" containerID="76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1" Jul 24 17:08:36.601443 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.601386 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" event={"ID":"202119ad-4258-4448-9210-a436ba05064a","Type":"ContainerStarted","Data":"72f8fbc27086a14be0024620080ce043e54d19aabc367fb7a4cea5fac4b737c0"} Jul 24 17:08:36.608600 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.608584 2552 scope.go:117] "RemoveContainer" containerID="76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1" Jul 24 17:08:36.608870 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:08:36.608847 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1\": container with ID starting with 76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1 not found: ID does not exist" containerID="76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1" Jul 24 17:08:36.608977 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.608879 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1"} err="failed to get container status \"76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1\": rpc error: code = NotFound desc = could not find container \"76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1\": container with ID starting with 76ef3034e97efb86d4e9b6eb60200ae59714d938df9b9cea8a3e1651bc2857d1 not found: ID does not exist" Jul 24 17:08:36.658026 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.657978 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" podStartSLOduration=2.362351206 podStartE2EDuration="5.657964457s" totalImagesPullingTime="3.295613251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:08:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:36.653092178 +0000 UTC m=+334.151494706" watchObservedRunningTime="2026-07-24 17:08:36.657964457 +0000 UTC m=+334.156367003" Jul 24 17:08:36.681384 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.681343 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br"] Jul 24 17:08:36.688332 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.688303 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-df8br"] Jul 24 17:08:36.966308 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:36.966265 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f79083a5-ee09-4586-ba1b-eb05e1e560dd" path="/var/lib/kubelet/pods/f79083a5-ee09-4586-ba1b-eb05e1e560dd/volumes" Jul 24 17:08:37.333301 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:37.333229 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:37.338239 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:37.338211 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:37.609207 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:37.609126 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:37.610306 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:37.610280 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v" Jul 24 17:08:38.029838 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:38.029800 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:08:39.625506 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:39.625468 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" event={"ID":"202119ad-4258-4448-9210-a436ba05064a","Type":"ContainerStarted","Data":"654b4ea12ab1bc1e98dfef10bce3ee849e0e261519b802dd9aaf43d7bf983786"} Jul 24 17:08:39.625938 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:39.625735 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:08:39.644480 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:39.644437 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" podStartSLOduration=2.837526594 podStartE2EDuration="5.644398899s" totalImagesPullingTime="2.806872305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:08:39.643384144 +0000 UTC m=+337.141786689" watchObservedRunningTime="2026-07-24 17:08:39.644398899 +0000 UTC m=+337.142801452" Jul 24 17:08:41.576608 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:41.576529 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9wwg9" Jul 24 17:08:41.617901 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:08:41.617868 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rs69p" Jul 24 17:09:10.632376 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:10.632343 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rtpmx" Jul 24 17:09:14.574958 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.574921 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:14.673439 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.673372 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:14.700025 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.699996 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4hgcd"] Jul 24 17:09:14.700337 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.700326 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f79083a5-ee09-4586-ba1b-eb05e1e560dd" containerName="discovery" Jul 24 17:09:14.700384 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.700338 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f79083a5-ee09-4586-ba1b-eb05e1e560dd" containerName="discovery" Jul 24 17:09:14.700456 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.700398 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f79083a5-ee09-4586-ba1b-eb05e1e560dd" containerName="discovery" Jul 24 17:09:14.711160 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.711132 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4hgcd"] Jul 24 17:09:14.711312 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.711227 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" Jul 24 17:09:14.714210 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.714178 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 17:09:14.714381 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.714355 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-5srk8\"" Jul 24 17:09:14.714510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.714221 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 17:09:14.772223 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.772182 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:14.807109 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.807076 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvw9\" (UniqueName: \"kubernetes.io/projected/b79b5fda-edad-4912-aabb-746c2c3ee6cd-kube-api-access-fjvw9\") pod \"kuadrant-operator-catalog-4hgcd\" (UID: \"b79b5fda-edad-4912-aabb-746c2c3ee6cd\") " pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" Jul 24 17:09:14.907659 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.907634 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvw9\" (UniqueName: \"kubernetes.io/projected/b79b5fda-edad-4912-aabb-746c2c3ee6cd-kube-api-access-fjvw9\") pod \"kuadrant-operator-catalog-4hgcd\" (UID: \"b79b5fda-edad-4912-aabb-746c2c3ee6cd\") " pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" Jul 24 17:09:14.915371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:14.915342 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvw9\" (UniqueName: \"kubernetes.io/projected/b79b5fda-edad-4912-aabb-746c2c3ee6cd-kube-api-access-fjvw9\") pod \"kuadrant-operator-catalog-4hgcd\" (UID: \"b79b5fda-edad-4912-aabb-746c2c3ee6cd\") " pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" Jul 24 17:09:15.022600 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.022565 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" Jul 24 17:09:15.072583 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.072554 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4hgcd"] Jul 24 17:09:15.149178 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.149152 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4hgcd"] Jul 24 17:09:15.151794 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:09:15.151770 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79b5fda_edad_4912_aabb_746c2c3ee6cd.slice/crio-2c755e32b53553cebb6f744f26df7c6ad77be88ddc5b35ff1c4aef9f812d69fb WatchSource:0}: Error finding container 2c755e32b53553cebb6f744f26df7c6ad77be88ddc5b35ff1c4aef9f812d69fb: Status 404 returned error can't find the container with id 2c755e32b53553cebb6f744f26df7c6ad77be88ddc5b35ff1c4aef9f812d69fb Jul 24 17:09:15.282017 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.281938 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq7tx"] Jul 24 17:09:15.284600 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.284585 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:15.292752 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.292723 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq7tx"] Jul 24 17:09:15.311781 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.311757 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7v45\" (UniqueName: \"kubernetes.io/projected/503b0002-85f0-453f-a628-deeec289abaa-kube-api-access-f7v45\") pod \"kuadrant-operator-catalog-dq7tx\" (UID: \"503b0002-85f0-453f-a628-deeec289abaa\") " pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:15.412231 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.412189 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7v45\" (UniqueName: \"kubernetes.io/projected/503b0002-85f0-453f-a628-deeec289abaa-kube-api-access-f7v45\") pod \"kuadrant-operator-catalog-dq7tx\" (UID: \"503b0002-85f0-453f-a628-deeec289abaa\") " pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:15.420687 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.420656 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7v45\" (UniqueName: \"kubernetes.io/projected/503b0002-85f0-453f-a628-deeec289abaa-kube-api-access-f7v45\") pod \"kuadrant-operator-catalog-dq7tx\" (UID: \"503b0002-85f0-453f-a628-deeec289abaa\") " pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:15.594697 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.594615 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:15.745019 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.744996 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq7tx"] Jul 24 17:09:15.746959 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:09:15.746937 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503b0002_85f0_453f_a628_deeec289abaa.slice/crio-9b7a612473855d9ea7a620588d1ccc4e99264e5cb8db8ac0d00d73b3c97eaf87 WatchSource:0}: Error finding container 9b7a612473855d9ea7a620588d1ccc4e99264e5cb8db8ac0d00d73b3c97eaf87: Status 404 returned error can't find the container with id 9b7a612473855d9ea7a620588d1ccc4e99264e5cb8db8ac0d00d73b3c97eaf87 Jul 24 17:09:15.785935 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.785893 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" event={"ID":"503b0002-85f0-453f-a628-deeec289abaa","Type":"ContainerStarted","Data":"9b7a612473855d9ea7a620588d1ccc4e99264e5cb8db8ac0d00d73b3c97eaf87"} Jul 24 17:09:15.787007 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:15.786982 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" event={"ID":"b79b5fda-edad-4912-aabb-746c2c3ee6cd","Type":"ContainerStarted","Data":"2c755e32b53553cebb6f744f26df7c6ad77be88ddc5b35ff1c4aef9f812d69fb"} Jul 24 17:09:17.798024 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:17.797924 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" event={"ID":"503b0002-85f0-453f-a628-deeec289abaa","Type":"ContainerStarted","Data":"5e1b545024175204ffbee1a49f2ae4a5b1ba7f3f5ac0140a05c327957c8c4157"} Jul 24 17:09:17.799343 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:17.799324 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" event={"ID":"b79b5fda-edad-4912-aabb-746c2c3ee6cd","Type":"ContainerStarted","Data":"13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba"} Jul 24 17:09:17.799430 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:17.799355 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" podUID="b79b5fda-edad-4912-aabb-746c2c3ee6cd" containerName="registry-server" containerID="cri-o://13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba" gracePeriod=2 Jul 24 17:09:17.813842 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:17.813798 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" podStartSLOduration=1.128218255 podStartE2EDuration="2.81378144s" totalImagesPullingTime="1.685563185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:09:17.81185317 +0000 UTC m=+375.310255716" watchObservedRunningTime="2026-07-24 17:09:17.81378144 +0000 UTC m=+375.312183986" Jul 24 17:09:17.827036 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:17.826996 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" podStartSLOduration=1.5492231109999999 podStartE2EDuration="3.826982211s" totalImagesPullingTime="2.2777591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:09:17.825070289 +0000 UTC m=+375.323472834" watchObservedRunningTime="2026-07-24 17:09:17.826982211 +0000 UTC m=+375.325384770" Jul 24 17:09:18.041761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.041737 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" Jul 24 17:09:18.136353 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.136270 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjvw9\" (UniqueName: \"kubernetes.io/projected/b79b5fda-edad-4912-aabb-746c2c3ee6cd-kube-api-access-fjvw9\") pod \"b79b5fda-edad-4912-aabb-746c2c3ee6cd\" (UID: \"b79b5fda-edad-4912-aabb-746c2c3ee6cd\") " Jul 24 17:09:18.138393 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.138357 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79b5fda-edad-4912-aabb-746c2c3ee6cd-kube-api-access-fjvw9" (OuterVolumeSpecName: "kube-api-access-fjvw9") pod "b79b5fda-edad-4912-aabb-746c2c3ee6cd" (UID: "b79b5fda-edad-4912-aabb-746c2c3ee6cd"). InnerVolumeSpecName "kube-api-access-fjvw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:09:18.236991 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.236941 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjvw9\" (UniqueName: \"kubernetes.io/projected/b79b5fda-edad-4912-aabb-746c2c3ee6cd-kube-api-access-fjvw9\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:09:18.804375 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.804289 2552 generic.go:356] "Generic (PLEG): container finished" podID="b79b5fda-edad-4912-aabb-746c2c3ee6cd" containerID="13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba" exitCode=0 Jul 24 17:09:18.804375 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.804329 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" Jul 24 17:09:18.804375 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.804365 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" event={"ID":"b79b5fda-edad-4912-aabb-746c2c3ee6cd","Type":"ContainerDied","Data":"13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba"} Jul 24 17:09:18.804953 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.804430 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4hgcd" event={"ID":"b79b5fda-edad-4912-aabb-746c2c3ee6cd","Type":"ContainerDied","Data":"2c755e32b53553cebb6f744f26df7c6ad77be88ddc5b35ff1c4aef9f812d69fb"} Jul 24 17:09:18.804953 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.804456 2552 scope.go:117] "RemoveContainer" containerID="13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba" Jul 24 17:09:18.813909 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.813893 2552 scope.go:117] "RemoveContainer" containerID="13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba" Jul 24 17:09:18.814154 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:09:18.814137 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba\": container with ID starting with 13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba not found: ID does not exist" containerID="13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba" Jul 24 17:09:18.814212 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.814162 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba"} err="failed to get container status \"13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba\": rpc error: code = NotFound desc = could not find container \"13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba\": container with ID starting with 13c7dc40e89ddd858cfa97be23e9ad2e7d7c04ffada88cc035a6b82c83a665ba not found: ID does not exist" Jul 24 17:09:18.824327 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.824305 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4hgcd"] Jul 24 17:09:18.826511 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.826492 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4hgcd"] Jul 24 17:09:18.963477 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:18.963447 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b79b5fda-edad-4912-aabb-746c2c3ee6cd" path="/var/lib/kubelet/pods/b79b5fda-edad-4912-aabb-746c2c3ee6cd/volumes" Jul 24 17:09:25.594896 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:25.594862 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:25.595402 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:25.594931 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:25.616135 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:25.616106 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:25.858147 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:25.858080 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-dq7tx" Jul 24 17:09:37.540861 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:37.540830 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:38.143094 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:38.143053 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:38.673265 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:38.673232 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:39.377582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:39.377543 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:41.422594 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.422561 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7pks2"] Jul 24 17:09:41.422956 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.422914 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b79b5fda-edad-4912-aabb-746c2c3ee6cd" containerName="registry-server" Jul 24 17:09:41.422956 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.422928 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b79b5fda-edad-4912-aabb-746c2c3ee6cd" containerName="registry-server" Jul 24 17:09:41.423026 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.423004 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b79b5fda-edad-4912-aabb-746c2c3ee6cd" containerName="registry-server" Jul 24 17:09:41.426281 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.426261 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" Jul 24 17:09:41.428917 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.428899 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9c6m5\"" Jul 24 17:09:41.443024 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.443001 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7pks2"] Jul 24 17:09:41.533343 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.533306 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqqq\" (UniqueName: \"kubernetes.io/projected/480bfbf3-8300-420a-b091-f34e5118bbfa-kube-api-access-bmqqq\") pod \"authorino-operator-75cc8d5cb7-7pks2\" (UID: \"480bfbf3-8300-420a-b091-f34e5118bbfa\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" Jul 24 17:09:41.633796 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.633709 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqqq\" (UniqueName: \"kubernetes.io/projected/480bfbf3-8300-420a-b091-f34e5118bbfa-kube-api-access-bmqqq\") pod \"authorino-operator-75cc8d5cb7-7pks2\" (UID: \"480bfbf3-8300-420a-b091-f34e5118bbfa\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" Jul 24 17:09:41.645303 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.645279 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqqq\" (UniqueName: \"kubernetes.io/projected/480bfbf3-8300-420a-b091-f34e5118bbfa-kube-api-access-bmqqq\") pod \"authorino-operator-75cc8d5cb7-7pks2\" (UID: \"480bfbf3-8300-420a-b091-f34e5118bbfa\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" Jul 24 17:09:41.736602 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.736577 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" Jul 24 17:09:41.871992 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.871966 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7pks2"] Jul 24 17:09:41.873930 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:09:41.873902 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480bfbf3_8300_420a_b091_f34e5118bbfa.slice/crio-50707a2c22b9d898f9d90205d4fc1a44569435c42b559848b77be0b3128c7570 WatchSource:0}: Error finding container 50707a2c22b9d898f9d90205d4fc1a44569435c42b559848b77be0b3128c7570: Status 404 returned error can't find the container with id 50707a2c22b9d898f9d90205d4fc1a44569435c42b559848b77be0b3128c7570 Jul 24 17:09:41.912020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:41.911993 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" event={"ID":"480bfbf3-8300-420a-b091-f34e5118bbfa","Type":"ContainerStarted","Data":"50707a2c22b9d898f9d90205d4fc1a44569435c42b559848b77be0b3128c7570"} Jul 24 17:09:42.165980 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:42.165904 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:42.388689 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:42.388652 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:44.928990 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:44.928958 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" event={"ID":"480bfbf3-8300-420a-b091-f34e5118bbfa","Type":"ContainerStarted","Data":"51fce558fa83cd732a87071eeb5c2806de7e0c6d8659861d251e809916c08b04"} Jul 24 17:09:44.929440 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:44.929115 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" Jul 24 17:09:44.953838 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:44.953784 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" podStartSLOduration=1.716725177 podStartE2EDuration="3.953768774s" totalImagesPullingTime="2.237043597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:09:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:09:44.950590438 +0000 UTC m=+402.448992981" watchObservedRunningTime="2026-07-24 17:09:44.953768774 +0000 UTC m=+402.452171320" Jul 24 17:09:45.189821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.189746 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k"] Jul 24 17:09:45.193129 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.193113 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.195656 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.195634 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wjdgd\"" Jul 24 17:09:45.207000 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.206980 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k"] Jul 24 17:09:45.267707 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.267671 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1340e7d2-8325-430f-a022-f6b3613d61a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.267848 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.267791 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46kp\" (UniqueName: \"kubernetes.io/projected/1340e7d2-8325-430f-a022-f6b3613d61a7-kube-api-access-v46kp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.368349 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.368310 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1340e7d2-8325-430f-a022-f6b3613d61a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.368545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.368386 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46kp\" (UniqueName: \"kubernetes.io/projected/1340e7d2-8325-430f-a022-f6b3613d61a7-kube-api-access-v46kp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.368710 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.368691 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1340e7d2-8325-430f-a022-f6b3613d61a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.379199 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.379171 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46kp\" (UniqueName: \"kubernetes.io/projected/1340e7d2-8325-430f-a022-f6b3613d61a7-kube-api-access-v46kp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.503396 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.503318 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:45.639621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.639597 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k"] Jul 24 17:09:45.641716 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:09:45.641687 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1340e7d2_8325_430f_a022_f6b3613d61a7.slice/crio-6693218265a486c662fb347f305e323f0268f3a1d550db279d7be03bd5762236 WatchSource:0}: Error finding container 6693218265a486c662fb347f305e323f0268f3a1d550db279d7be03bd5762236: Status 404 returned error can't find the container with id 6693218265a486c662fb347f305e323f0268f3a1d550db279d7be03bd5762236 Jul 24 17:09:45.767949 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.767866 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:45.934640 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:45.934604 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" event={"ID":"1340e7d2-8325-430f-a022-f6b3613d61a7","Type":"ContainerStarted","Data":"6693218265a486c662fb347f305e323f0268f3a1d550db279d7be03bd5762236"} Jul 24 17:09:47.206584 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.206548 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss"] Jul 24 17:09:47.210341 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.210324 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" Jul 24 17:09:47.212763 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.212739 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 17:09:47.212858 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.212840 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-vtcsf\"" Jul 24 17:09:47.219071 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.219046 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss"] Jul 24 17:09:47.283998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.283965 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69578\" (UniqueName: \"kubernetes.io/projected/4154d801-e622-49fb-8ec7-6d231864e233-kube-api-access-69578\") pod \"dns-operator-controller-manager-864dd78c4c-q7kss\" (UID: \"4154d801-e622-49fb-8ec7-6d231864e233\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" Jul 24 17:09:47.385044 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.385004 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69578\" (UniqueName: \"kubernetes.io/projected/4154d801-e622-49fb-8ec7-6d231864e233-kube-api-access-69578\") pod \"dns-operator-controller-manager-864dd78c4c-q7kss\" (UID: \"4154d801-e622-49fb-8ec7-6d231864e233\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" Jul 24 17:09:47.395595 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.395565 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69578\" (UniqueName: \"kubernetes.io/projected/4154d801-e622-49fb-8ec7-6d231864e233-kube-api-access-69578\") pod \"dns-operator-controller-manager-864dd78c4c-q7kss\" (UID: \"4154d801-e622-49fb-8ec7-6d231864e233\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" Jul 24 17:09:47.521266 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.521180 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" Jul 24 17:09:47.667459 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.667436 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss"] Jul 24 17:09:47.669678 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:09:47.669649 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4154d801_e622_49fb_8ec7_6d231864e233.slice/crio-03a57bbbee226190095966a9296f7895eabb4d233eb6bf88b34ecd2be7f11b34 WatchSource:0}: Error finding container 03a57bbbee226190095966a9296f7895eabb4d233eb6bf88b34ecd2be7f11b34: Status 404 returned error can't find the container with id 03a57bbbee226190095966a9296f7895eabb4d233eb6bf88b34ecd2be7f11b34 Jul 24 17:09:47.753206 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.753129 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:47.947689 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:47.947654 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" event={"ID":"4154d801-e622-49fb-8ec7-6d231864e233","Type":"ContainerStarted","Data":"03a57bbbee226190095966a9296f7895eabb4d233eb6bf88b34ecd2be7f11b34"} Jul 24 17:09:48.105248 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:48.105213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:49.805883 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.805842 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk"] Jul 24 17:09:49.809627 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.809598 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:09:49.811817 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.811793 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kzfk2\"" Jul 24 17:09:49.818177 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.817895 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk"] Jul 24 17:09:49.906382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.906357 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7v85\" (UniqueName: \"kubernetes.io/projected/6bbf2a2c-7486-4b27-8bab-4fb238b8278d-kube-api-access-z7v85\") pod \"limitador-operator-controller-manager-859d7b78bd-q8djk\" (UID: \"6bbf2a2c-7486-4b27-8bab-4fb238b8278d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:09:49.959373 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.959340 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" event={"ID":"4154d801-e622-49fb-8ec7-6d231864e233","Type":"ContainerStarted","Data":"b76d5a6ca17f3f09762fa32f44ffee07a980b0019c91dae39caccafd99d6fb76"} Jul 24 17:09:49.959532 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.959432 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" Jul 24 17:09:49.978816 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:49.978764 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" podStartSLOduration=1.000082924 podStartE2EDuration="2.978748306s" totalImagesPullingTime="1.978665382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:09:49.976851149 +0000 UTC m=+407.475253696" watchObservedRunningTime="2026-07-24 17:09:49.978748306 +0000 UTC m=+407.477150852" Jul 24 17:09:50.007768 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:50.007734 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7v85\" (UniqueName: \"kubernetes.io/projected/6bbf2a2c-7486-4b27-8bab-4fb238b8278d-kube-api-access-z7v85\") pod \"limitador-operator-controller-manager-859d7b78bd-q8djk\" (UID: \"6bbf2a2c-7486-4b27-8bab-4fb238b8278d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:09:50.016639 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:50.016614 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7v85\" (UniqueName: \"kubernetes.io/projected/6bbf2a2c-7486-4b27-8bab-4fb238b8278d-kube-api-access-z7v85\") pod \"limitador-operator-controller-manager-859d7b78bd-q8djk\" (UID: \"6bbf2a2c-7486-4b27-8bab-4fb238b8278d\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:09:50.122543 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:50.122511 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:09:50.385217 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:50.385130 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:51.253343 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.253318 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk"] Jul 24 17:09:51.254786 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:09:51.254759 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbf2a2c_7486_4b27_8bab_4fb238b8278d.slice/crio-74e97295fa03422c1fcb63897ca73863e3b71c51336bb6800f47a32c70b02494 WatchSource:0}: Error finding container 74e97295fa03422c1fcb63897ca73863e3b71c51336bb6800f47a32c70b02494: Status 404 returned error can't find the container with id 74e97295fa03422c1fcb63897ca73863e3b71c51336bb6800f47a32c70b02494 Jul 24 17:09:51.548935 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.548906 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:51.608162 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.608132 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:51.950864 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.950830 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:51.971728 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.971690 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" event={"ID":"1340e7d2-8325-430f-a022-f6b3613d61a7","Type":"ContainerStarted","Data":"5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8"} Jul 24 17:09:51.971891 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.971753 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:09:51.973004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.972980 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" event={"ID":"6bbf2a2c-7486-4b27-8bab-4fb238b8278d","Type":"ContainerStarted","Data":"74e97295fa03422c1fcb63897ca73863e3b71c51336bb6800f47a32c70b02494"} Jul 24 17:09:51.998525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:51.998480 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" podStartSLOduration=1.45081812 podStartE2EDuration="6.998463658s" totalImagesPullingTime="5.547645538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:09:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:09:51.995916226 +0000 UTC m=+409.494318767" watchObservedRunningTime="2026-07-24 17:09:51.998463658 +0000 UTC m=+409.496866271" Jul 24 17:09:52.051019 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:52.050988 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:53.823482 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:53.823445 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:53.930261 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:53.930228 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:53.984921 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:53.984886 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" event={"ID":"6bbf2a2c-7486-4b27-8bab-4fb238b8278d","Type":"ContainerStarted","Data":"20d3a76944af5407e0dae5c3242233fbd507b4052ea3feb2b3292442dc29b554"} Jul 24 17:09:53.985101 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:53.985008 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:09:54.000341 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:54.000291 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" podStartSLOduration=3.062906305 podStartE2EDuration="5.000273532s" totalImagesPullingTime="1.937367227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:09:53.999276926 +0000 UTC m=+411.497679485" watchObservedRunningTime="2026-07-24 17:09:54.000273532 +0000 UTC m=+411.498676096" Jul 24 17:09:54.003565 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:54.003538 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:54.127821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:54.127728 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:55.550759 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:55.550725 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:55.610767 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:55.610738 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:55.937177 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:55.937146 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7pks2" Jul 24 17:09:59.164750 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:59.164706 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:09:59.265751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:09:59.265715 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:00.967531 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:00.967504 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q7kss" Jul 24 17:10:02.559996 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:02.559958 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:02.623447 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:02.623388 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:02.980419 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:02.980391 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:10:04.753840 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.753807 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k"] Jul 24 17:10:04.754244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.754012 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" containerName="manager" containerID="cri-o://5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8" gracePeriod=2 Jul 24 17:10:04.764638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.764609 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k"] Jul 24 17:10:04.775617 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.775590 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk"] Jul 24 17:10:04.775893 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.775843 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" containerName="manager" containerID="cri-o://20d3a76944af5407e0dae5c3242233fbd507b4052ea3feb2b3292442dc29b554" gracePeriod=2 Jul 24 17:10:04.777787 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:04.777753 2552 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1340e7d2-8325-430f-a022-f6b3613d61a7\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\": pods \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:10:04.778018 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.777948 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:10:04.779867 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.779844 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz"] Jul 24 17:10:04.780369 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.780348 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" containerName="manager" Jul 24 17:10:04.780473 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.780370 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" containerName="manager" Jul 24 17:10:04.780545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.780502 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" containerName="manager" Jul 24 17:10:04.781284 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:04.781250 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" is forbidden: User \"system:node:ip-10-0-138-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-185.ec2.internal' and this object" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:10:04.783397 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:04.783368 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8djk\" is forbidden: User \"system:node:ip-10-0-138-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-185.ec2.internal' and this object" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:10:04.783670 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.783648 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk"] Jul 24 17:10:04.783752 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.783738 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:04.786750 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:04.786720 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" is forbidden: User \"system:node:ip-10-0-138-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-185.ec2.internal' and this object" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:10:04.788454 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:04.788400 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8djk\" is forbidden: User \"system:node:ip-10-0-138-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-185.ec2.internal' and this object" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:10:04.801126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.801100 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz"] Jul 24 17:10:04.810782 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.810758 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4"] Jul 24 17:10:04.811161 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.811144 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" containerName="manager" Jul 24 17:10:04.811161 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.811161 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" containerName="manager" Jul 24 17:10:04.811306 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.811225 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" containerName="manager" Jul 24 17:10:04.814132 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.814115 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" Jul 24 17:10:04.817911 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:04.817889 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bmx4k\" is forbidden: User \"system:node:ip-10-0-138-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-185.ec2.internal' and this object" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:10:04.823921 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.823899 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4"] Jul 24 17:10:04.837803 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:04.837770 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8djk\" is forbidden: User \"system:node:ip-10-0-138-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-185.ec2.internal' and this object" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:10:04.936841 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.936702 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59cb50db-2ff7-4518-b03e-45dd547ba0b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zn4pz\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:04.936841 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.936748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7w85\" (UniqueName: \"kubernetes.io/projected/029eaa23-1a41-4924-ab23-74d0b999d8ff-kube-api-access-l7w85\") pod \"limitador-operator-controller-manager-859d7b78bd-gllt4\" (UID: \"029eaa23-1a41-4924-ab23-74d0b999d8ff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" Jul 24 17:10:04.936841 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:04.936808 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx8jh\" (UniqueName: \"kubernetes.io/projected/59cb50db-2ff7-4518-b03e-45dd547ba0b2-kube-api-access-rx8jh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zn4pz\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:05.013179 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.013117 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:10:05.037925 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.037887 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx8jh\" (UniqueName: \"kubernetes.io/projected/59cb50db-2ff7-4518-b03e-45dd547ba0b2-kube-api-access-rx8jh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zn4pz\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:05.038083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.038020 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59cb50db-2ff7-4518-b03e-45dd547ba0b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zn4pz\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:05.038083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.038065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7w85\" (UniqueName: \"kubernetes.io/projected/029eaa23-1a41-4924-ab23-74d0b999d8ff-kube-api-access-l7w85\") pod \"limitador-operator-controller-manager-859d7b78bd-gllt4\" (UID: \"029eaa23-1a41-4924-ab23-74d0b999d8ff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" Jul 24 17:10:05.038446 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.038397 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59cb50db-2ff7-4518-b03e-45dd547ba0b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zn4pz\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:05.040368 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.040344 2552 generic.go:356] "Generic (PLEG): container finished" podID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" containerID="20d3a76944af5407e0dae5c3242233fbd507b4052ea3feb2b3292442dc29b554" exitCode=0 Jul 24 17:10:05.040496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.040438 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e97295fa03422c1fcb63897ca73863e3b71c51336bb6800f47a32c70b02494" Jul 24 17:10:05.041928 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.041872 2552 generic.go:356] "Generic (PLEG): container finished" podID="1340e7d2-8325-430f-a022-f6b3613d61a7" containerID="5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8" exitCode=0 Jul 24 17:10:05.041928 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.041922 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bmx4k" Jul 24 17:10:05.042113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.041938 2552 scope.go:117] "RemoveContainer" containerID="5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8" Jul 24 17:10:05.043000 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.042976 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:10:05.051273 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.051240 2552 scope.go:117] "RemoveContainer" containerID="5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8" Jul 24 17:10:05.051273 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.051266 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx8jh\" (UniqueName: \"kubernetes.io/projected/59cb50db-2ff7-4518-b03e-45dd547ba0b2-kube-api-access-rx8jh\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zn4pz\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:05.051625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.051599 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7w85\" (UniqueName: \"kubernetes.io/projected/029eaa23-1a41-4924-ab23-74d0b999d8ff-kube-api-access-l7w85\") pod \"limitador-operator-controller-manager-859d7b78bd-gllt4\" (UID: \"029eaa23-1a41-4924-ab23-74d0b999d8ff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" Jul 24 17:10:05.051851 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:05.051826 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8\": container with ID starting with 5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8 not found: ID does not exist" containerID="5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8" Jul 24 17:10:05.051928 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.051860 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8"} err="failed to get container status \"5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8\": rpc error: code = NotFound desc = could not find container \"5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8\": container with ID starting with 5bfe9f905e2563b9611852062642510f80912b2072d6fe33d2dc0581b4b014b8 not found: ID does not exist" Jul 24 17:10:05.139394 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.139368 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1340e7d2-8325-430f-a022-f6b3613d61a7-extensions-socket-volume\") pod \"1340e7d2-8325-430f-a022-f6b3613d61a7\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " Jul 24 17:10:05.139584 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.139466 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v46kp\" (UniqueName: \"kubernetes.io/projected/1340e7d2-8325-430f-a022-f6b3613d61a7-kube-api-access-v46kp\") pod \"1340e7d2-8325-430f-a022-f6b3613d61a7\" (UID: \"1340e7d2-8325-430f-a022-f6b3613d61a7\") " Jul 24 17:10:05.139754 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.139727 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1340e7d2-8325-430f-a022-f6b3613d61a7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1340e7d2-8325-430f-a022-f6b3613d61a7" (UID: "1340e7d2-8325-430f-a022-f6b3613d61a7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:10:05.141582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.141550 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:05.141867 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.141842 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1340e7d2-8325-430f-a022-f6b3613d61a7-kube-api-access-v46kp" (OuterVolumeSpecName: "kube-api-access-v46kp") pod "1340e7d2-8325-430f-a022-f6b3613d61a7" (UID: "1340e7d2-8325-430f-a022-f6b3613d61a7"). InnerVolumeSpecName "kube-api-access-v46kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:10:05.199007 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.198984 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:05.205252 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.205235 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" Jul 24 17:10:05.240786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.240749 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7v85\" (UniqueName: \"kubernetes.io/projected/6bbf2a2c-7486-4b27-8bab-4fb238b8278d-kube-api-access-z7v85\") pod \"6bbf2a2c-7486-4b27-8bab-4fb238b8278d\" (UID: \"6bbf2a2c-7486-4b27-8bab-4fb238b8278d\") " Jul 24 17:10:05.241124 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.241102 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1340e7d2-8325-430f-a022-f6b3613d61a7-extensions-socket-volume\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:10:05.241222 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.241127 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46kp\" (UniqueName: \"kubernetes.io/projected/1340e7d2-8325-430f-a022-f6b3613d61a7-kube-api-access-v46kp\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:10:05.243638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.243609 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbf2a2c-7486-4b27-8bab-4fb238b8278d-kube-api-access-z7v85" (OuterVolumeSpecName: "kube-api-access-z7v85") pod "6bbf2a2c-7486-4b27-8bab-4fb238b8278d" (UID: "6bbf2a2c-7486-4b27-8bab-4fb238b8278d"). InnerVolumeSpecName "kube-api-access-z7v85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:10:05.246926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.246889 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:05.342040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.342012 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz"] Jul 24 17:10:05.342385 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.342360 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7v85\" (UniqueName: \"kubernetes.io/projected/6bbf2a2c-7486-4b27-8bab-4fb238b8278d-kube-api-access-z7v85\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:10:05.344104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.344075 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:05.344651 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:10:05.344629 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59cb50db_2ff7_4518_b03e_45dd547ba0b2.slice/crio-f0aa295ed30c810ca9316482d386eedf9cb4b92cda639e0a3109901396b7ae8f WatchSource:0}: Error finding container f0aa295ed30c810ca9316482d386eedf9cb4b92cda639e0a3109901396b7ae8f: Status 404 returned error can't find the container with id f0aa295ed30c810ca9316482d386eedf9cb4b92cda639e0a3109901396b7ae8f Jul 24 17:10:05.356633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:05.356615 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4"] Jul 24 17:10:05.360510 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:10:05.360480 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029eaa23_1a41_4924_ab23_74d0b999d8ff.slice/crio-b85bd004a63ebe5608a7413ba4c406b8f402370c593c95b368369f869e5080e4 WatchSource:0}: Error finding container b85bd004a63ebe5608a7413ba4c406b8f402370c593c95b368369f869e5080e4: Status 404 returned error can't find the container with id b85bd004a63ebe5608a7413ba4c406b8f402370c593c95b368369f869e5080e4 Jul 24 17:10:06.048546 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.048447 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" event={"ID":"029eaa23-1a41-4924-ab23-74d0b999d8ff","Type":"ContainerStarted","Data":"c388d127c269e5b7252bf0430d83535ea3b849cdccd4d545c243ce4eb80aa745"} Jul 24 17:10:06.048964 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.048599 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" Jul 24 17:10:06.048964 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.048642 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" event={"ID":"029eaa23-1a41-4924-ab23-74d0b999d8ff","Type":"ContainerStarted","Data":"b85bd004a63ebe5608a7413ba4c406b8f402370c593c95b368369f869e5080e4"} Jul 24 17:10:06.051194 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.051168 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:10:06.051332 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.051170 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" event={"ID":"59cb50db-2ff7-4518-b03e-45dd547ba0b2","Type":"ContainerStarted","Data":"1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c"} Jul 24 17:10:06.051332 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.051290 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" event={"ID":"59cb50db-2ff7-4518-b03e-45dd547ba0b2","Type":"ContainerStarted","Data":"f0aa295ed30c810ca9316482d386eedf9cb4b92cda639e0a3109901396b7ae8f"} Jul 24 17:10:06.051332 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.051321 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:06.065529 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.065471 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" podStartSLOduration=2.065458764 podStartE2EDuration="2.065458764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:10:06.064833232 +0000 UTC m=+423.563235813" watchObservedRunningTime="2026-07-24 17:10:06.065458764 +0000 UTC m=+423.563861311" Jul 24 17:10:06.066949 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:06.066926 2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8djk\" is forbidden: User \"system:node:ip-10-0-138-185.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-185.ec2.internal' and this object" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8djk" Jul 24 17:10:06.088735 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.088689 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" podStartSLOduration=2.088672137 podStartE2EDuration="2.088672137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:10:06.088312878 +0000 UTC m=+423.586715425" watchObservedRunningTime="2026-07-24 17:10:06.088672137 +0000 UTC m=+423.587074688" Jul 24 17:10:06.964687 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.964657 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1340e7d2-8325-430f-a022-f6b3613d61a7" path="/var/lib/kubelet/pods/1340e7d2-8325-430f-a022-f6b3613d61a7/volumes" Jul 24 17:10:06.965005 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:06.964992 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bbf2a2c-7486-4b27-8bab-4fb238b8278d" path="/var/lib/kubelet/pods/6bbf2a2c-7486-4b27-8bab-4fb238b8278d/volumes" Jul 24 17:10:07.175676 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:07.175641 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:07.277801 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:07.277715 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:11.001123 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:11.001074 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:11.102728 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:11.102695 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:11.201590 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:11.201553 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:11.270751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:11.270658 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:13.688379 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:13.688340 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:13.789090 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:13.789035 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:17.057816 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:17.057785 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:17.058255 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:17.057954 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gllt4" Jul 24 17:10:20.210137 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:20.210104 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:20.309874 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:20.309835 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:25.014956 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:25.014924 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:25.096303 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:25.096275 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:28.736420 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:28.736376 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:28.789470 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:28.789443 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:29.335388 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:29.335355 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:29.402788 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:29.402757 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:33.133991 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:33.133957 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:33.187729 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:33.187690 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:36.181921 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.181885 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz"] Jul 24 17:10:36.182292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.182105 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" podUID="59cb50db-2ff7-4518-b03e-45dd547ba0b2" containerName="manager" containerID="cri-o://1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c" gracePeriod=10 Jul 24 17:10:36.435053 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.434992 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:36.485489 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.485459 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf"] Jul 24 17:10:36.485837 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.485825 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59cb50db-2ff7-4518-b03e-45dd547ba0b2" containerName="manager" Jul 24 17:10:36.485880 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.485839 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="59cb50db-2ff7-4518-b03e-45dd547ba0b2" containerName="manager" Jul 24 17:10:36.485924 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.485915 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="59cb50db-2ff7-4518-b03e-45dd547ba0b2" containerName="manager" Jul 24 17:10:36.487772 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.487755 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.489918 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.489898 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59cb50db-2ff7-4518-b03e-45dd547ba0b2-extensions-socket-volume\") pod \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " Jul 24 17:10:36.490005 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.489990 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx8jh\" (UniqueName: \"kubernetes.io/projected/59cb50db-2ff7-4518-b03e-45dd547ba0b2-kube-api-access-rx8jh\") pod \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\" (UID: \"59cb50db-2ff7-4518-b03e-45dd547ba0b2\") " Jul 24 17:10:36.490293 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.490267 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59cb50db-2ff7-4518-b03e-45dd547ba0b2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "59cb50db-2ff7-4518-b03e-45dd547ba0b2" (UID: "59cb50db-2ff7-4518-b03e-45dd547ba0b2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:10:36.491958 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.491927 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59cb50db-2ff7-4518-b03e-45dd547ba0b2-kube-api-access-rx8jh" (OuterVolumeSpecName: "kube-api-access-rx8jh") pod "59cb50db-2ff7-4518-b03e-45dd547ba0b2" (UID: "59cb50db-2ff7-4518-b03e-45dd547ba0b2"). InnerVolumeSpecName "kube-api-access-rx8jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:10:36.503791 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.503768 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf"] Jul 24 17:10:36.591402 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.591368 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66050e2-11ab-4070-aa52-9ad5aa4896d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hs6sf\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.591595 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.591514 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6jt\" (UniqueName: \"kubernetes.io/projected/c66050e2-11ab-4070-aa52-9ad5aa4896d2-kube-api-access-xf6jt\") pod \"kuadrant-operator-controller-manager-86878c769f-hs6sf\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.591652 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.591604 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx8jh\" (UniqueName: \"kubernetes.io/projected/59cb50db-2ff7-4518-b03e-45dd547ba0b2-kube-api-access-rx8jh\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:10:36.591652 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.591618 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/59cb50db-2ff7-4518-b03e-45dd547ba0b2-extensions-socket-volume\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:10:36.692046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.691962 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66050e2-11ab-4070-aa52-9ad5aa4896d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hs6sf\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.692202 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.692044 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6jt\" (UniqueName: \"kubernetes.io/projected/c66050e2-11ab-4070-aa52-9ad5aa4896d2-kube-api-access-xf6jt\") pod \"kuadrant-operator-controller-manager-86878c769f-hs6sf\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.692327 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.692304 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66050e2-11ab-4070-aa52-9ad5aa4896d2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hs6sf\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.701028 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.701008 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6jt\" (UniqueName: \"kubernetes.io/projected/c66050e2-11ab-4070-aa52-9ad5aa4896d2-kube-api-access-xf6jt\") pod \"kuadrant-operator-controller-manager-86878c769f-hs6sf\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.805552 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.805520 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:36.939171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:36.939147 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf"] Jul 24 17:10:36.942004 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:10:36.941978 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66050e2_11ab_4070_aa52_9ad5aa4896d2.slice/crio-e9c2b53592eb3cf18faaf5bd6e383eee2d6bfed0ae3e8fbfe8b9ccf480355484 WatchSource:0}: Error finding container e9c2b53592eb3cf18faaf5bd6e383eee2d6bfed0ae3e8fbfe8b9ccf480355484: Status 404 returned error can't find the container with id e9c2b53592eb3cf18faaf5bd6e383eee2d6bfed0ae3e8fbfe8b9ccf480355484 Jul 24 17:10:37.200951 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.200857 2552 generic.go:356] "Generic (PLEG): container finished" podID="59cb50db-2ff7-4518-b03e-45dd547ba0b2" containerID="1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c" exitCode=0 Jul 24 17:10:37.200951 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.200900 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" Jul 24 17:10:37.200951 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.200941 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" event={"ID":"59cb50db-2ff7-4518-b03e-45dd547ba0b2","Type":"ContainerDied","Data":"1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c"} Jul 24 17:10:37.201514 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.201005 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz" event={"ID":"59cb50db-2ff7-4518-b03e-45dd547ba0b2","Type":"ContainerDied","Data":"f0aa295ed30c810ca9316482d386eedf9cb4b92cda639e0a3109901396b7ae8f"} Jul 24 17:10:37.201514 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.201042 2552 scope.go:117] "RemoveContainer" containerID="1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c" Jul 24 17:10:37.203166 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.203137 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" event={"ID":"c66050e2-11ab-4070-aa52-9ad5aa4896d2","Type":"ContainerStarted","Data":"330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac"} Jul 24 17:10:37.203284 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.203188 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" event={"ID":"c66050e2-11ab-4070-aa52-9ad5aa4896d2","Type":"ContainerStarted","Data":"e9c2b53592eb3cf18faaf5bd6e383eee2d6bfed0ae3e8fbfe8b9ccf480355484"} Jul 24 17:10:37.203284 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.203223 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:37.210600 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.210585 2552 scope.go:117] "RemoveContainer" containerID="1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c" Jul 24 17:10:37.210823 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:10:37.210806 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c\": container with ID starting with 1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c not found: ID does not exist" containerID="1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c" Jul 24 17:10:37.210867 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.210832 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c"} err="failed to get container status \"1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c\": rpc error: code = NotFound desc = could not find container \"1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c\": container with ID starting with 1cbf74818c4e1f72c2139cb293b204d42e85269523fe760f1d1554873195933c not found: ID does not exist" Jul 24 17:10:37.221334 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.221287 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" podStartSLOduration=1.221274871 podStartE2EDuration="1.221274871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:10:37.220103753 +0000 UTC m=+454.718506298" watchObservedRunningTime="2026-07-24 17:10:37.221274871 +0000 UTC m=+454.719677416" Jul 24 17:10:37.236039 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.236014 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz"] Jul 24 17:10:37.240330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:37.240309 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zn4pz"] Jul 24 17:10:38.964233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:38.964205 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59cb50db-2ff7-4518-b03e-45dd547ba0b2" path="/var/lib/kubelet/pods/59cb50db-2ff7-4518-b03e-45dd547ba0b2/volumes" Jul 24 17:10:39.210087 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:39.210048 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:39.310881 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:39.310729 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:43.938341 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:43.938305 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:44.038807 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:44.038768 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:48.212361 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:48.212331 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:10:52.554222 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.554190 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t"] Jul 24 17:10:52.562161 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.562133 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.564498 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.564477 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kzlxc\"" Jul 24 17:10:52.569633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.569604 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t"] Jul 24 17:10:52.588442 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.588401 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:52.708851 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.708813 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:52.724927 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.724900 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.724935 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78961fb3-3403-4f44-a263-0e730fdb851a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.724957 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78961fb3-3403-4f44-a263-0e730fdb851a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.724976 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.724998 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.725015 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725051 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.725034 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78961fb3-3403-4f44-a263-0e730fdb851a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725282 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.725061 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.725282 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.725117 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpfq\" (UniqueName: \"kubernetes.io/projected/78961fb3-3403-4f44-a263-0e730fdb851a-kube-api-access-6rpfq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826169 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826091 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826169 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826131 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpfq\" (UniqueName: \"kubernetes.io/projected/78961fb3-3403-4f44-a263-0e730fdb851a-kube-api-access-6rpfq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826175 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826199 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78961fb3-3403-4f44-a263-0e730fdb851a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826214 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78961fb3-3403-4f44-a263-0e730fdb851a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826233 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826253 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826272 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.826382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826304 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78961fb3-3403-4f44-a263-0e730fdb851a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.827064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826590 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.827064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826605 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.827064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826716 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.827064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.826875 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.827064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.827056 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78961fb3-3403-4f44-a263-0e730fdb851a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.828630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.828607 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78961fb3-3403-4f44-a263-0e730fdb851a-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.828778 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.828757 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78961fb3-3403-4f44-a263-0e730fdb851a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.834123 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.834097 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpfq\" (UniqueName: \"kubernetes.io/projected/78961fb3-3403-4f44-a263-0e730fdb851a-kube-api-access-6rpfq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.834207 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.834121 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78961fb3-3403-4f44-a263-0e730fdb851a-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpt5t\" (UID: \"78961fb3-3403-4f44-a263-0e730fdb851a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:52.874426 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:52.874379 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:53.014173 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.014149 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t"] Jul 24 17:10:53.016495 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:10:53.016466 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78961fb3_3403_4f44_a263_0e730fdb851a.slice/crio-dd03faf8c0b443048c5cb48c6e705097b55d954dff5338081a7a9e52542fe17d WatchSource:0}: Error finding container dd03faf8c0b443048c5cb48c6e705097b55d954dff5338081a7a9e52542fe17d: Status 404 returned error can't find the container with id dd03faf8c0b443048c5cb48c6e705097b55d954dff5338081a7a9e52542fe17d Jul 24 17:10:53.019096 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.019067 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:10:53.019209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.019130 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:10:53.019209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.019162 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:10:53.283092 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.283057 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" event={"ID":"78961fb3-3403-4f44-a263-0e730fdb851a","Type":"ContainerStarted","Data":"6ee352c617bb303955160fa7c5bd22eef4f7bf377fb499212d8465ed766c5b81"} Jul 24 17:10:53.283292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.283123 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" event={"ID":"78961fb3-3403-4f44-a263-0e730fdb851a","Type":"ContainerStarted","Data":"dd03faf8c0b443048c5cb48c6e705097b55d954dff5338081a7a9e52542fe17d"} Jul 24 17:10:53.302992 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.302941 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" podStartSLOduration=1.302925583 podStartE2EDuration="1.302925583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:10:53.300995652 +0000 UTC m=+470.799398197" watchObservedRunningTime="2026-07-24 17:10:53.302925583 +0000 UTC m=+470.801328129" Jul 24 17:10:53.875153 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:53.875122 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:54.880026 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:54.879996 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:55.293658 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:55.293626 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:55.294525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:55.294508 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:10:55.669358 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:55.669326 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:55.768713 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:55.768675 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:55.871092 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:55.871002 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:56.509553 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.509520 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:56.570378 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.570343 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:10:56.670401 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.670368 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-sgvdq"] Jul 24 17:10:56.673712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.673697 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sgvdq" Jul 24 17:10:56.675930 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.675912 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vxxvf\"" Jul 24 17:10:56.681917 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.681897 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sgvdq"] Jul 24 17:10:56.762117 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.762022 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m296\" (UniqueName: \"kubernetes.io/projected/aa9640f3-0d6b-4f46-b3ce-21ef384415a8-kube-api-access-6m296\") pod \"authorino-75d556bd77-sgvdq\" (UID: \"aa9640f3-0d6b-4f46-b3ce-21ef384415a8\") " pod="kuadrant-system/authorino-75d556bd77-sgvdq" Jul 24 17:10:56.863586 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.863549 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m296\" (UniqueName: \"kubernetes.io/projected/aa9640f3-0d6b-4f46-b3ce-21ef384415a8-kube-api-access-6m296\") pod \"authorino-75d556bd77-sgvdq\" (UID: \"aa9640f3-0d6b-4f46-b3ce-21ef384415a8\") " pod="kuadrant-system/authorino-75d556bd77-sgvdq" Jul 24 17:10:56.871530 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.871503 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m296\" (UniqueName: \"kubernetes.io/projected/aa9640f3-0d6b-4f46-b3ce-21ef384415a8-kube-api-access-6m296\") pod \"authorino-75d556bd77-sgvdq\" (UID: \"aa9640f3-0d6b-4f46-b3ce-21ef384415a8\") " pod="kuadrant-system/authorino-75d556bd77-sgvdq" Jul 24 17:10:56.984425 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:56.984376 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sgvdq" Jul 24 17:10:57.111062 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:57.111036 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sgvdq"] Jul 24 17:10:57.112764 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:10:57.112732 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa9640f3_0d6b_4f46_b3ce_21ef384415a8.slice/crio-701901f27c50e36ec967dd0c498b9978f35cd971d7a73a40b45504b130de2a94 WatchSource:0}: Error finding container 701901f27c50e36ec967dd0c498b9978f35cd971d7a73a40b45504b130de2a94: Status 404 returned error can't find the container with id 701901f27c50e36ec967dd0c498b9978f35cd971d7a73a40b45504b130de2a94 Jul 24 17:10:57.303308 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:10:57.303220 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sgvdq" event={"ID":"aa9640f3-0d6b-4f46-b3ce-21ef384415a8","Type":"ContainerStarted","Data":"701901f27c50e36ec967dd0c498b9978f35cd971d7a73a40b45504b130de2a94"} Jul 24 17:11:00.324873 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:00.324833 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sgvdq" event={"ID":"aa9640f3-0d6b-4f46-b3ce-21ef384415a8","Type":"ContainerStarted","Data":"3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1"} Jul 24 17:11:00.340719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:00.340677 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-sgvdq" podStartSLOduration=1.304895868 podStartE2EDuration="4.340663447s" totalImagesPullingTime="3.035767579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:10:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:11:00.339829665 +0000 UTC m=+477.838232211" watchObservedRunningTime="2026-07-24 17:11:00.340663447 +0000 UTC m=+477.839065992" Jul 24 17:11:15.598449 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:15.598400 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:15.691432 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:15.691370 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:16.096120 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:16.096081 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:28.707036 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:28.706994 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:29.011735 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:29.011650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:30.476755 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:30.476710 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sgvdq"] Jul 24 17:11:30.477309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:30.476906 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-sgvdq" podUID="aa9640f3-0d6b-4f46-b3ce-21ef384415a8" containerName="authorino" containerID="cri-o://3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1" gracePeriod=30 Jul 24 17:11:30.748788 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:30.748763 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sgvdq" Jul 24 17:11:30.841608 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:30.841575 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m296\" (UniqueName: \"kubernetes.io/projected/aa9640f3-0d6b-4f46-b3ce-21ef384415a8-kube-api-access-6m296\") pod \"aa9640f3-0d6b-4f46-b3ce-21ef384415a8\" (UID: \"aa9640f3-0d6b-4f46-b3ce-21ef384415a8\") " Jul 24 17:11:30.843561 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:30.843535 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9640f3-0d6b-4f46-b3ce-21ef384415a8-kube-api-access-6m296" (OuterVolumeSpecName: "kube-api-access-6m296") pod "aa9640f3-0d6b-4f46-b3ce-21ef384415a8" (UID: "aa9640f3-0d6b-4f46-b3ce-21ef384415a8"). InnerVolumeSpecName "kube-api-access-6m296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:11:30.942938 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:30.942906 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m296\" (UniqueName: \"kubernetes.io/projected/aa9640f3-0d6b-4f46-b3ce-21ef384415a8-kube-api-access-6m296\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:11:31.482089 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.482052 2552 generic.go:356] "Generic (PLEG): container finished" podID="aa9640f3-0d6b-4f46-b3ce-21ef384415a8" containerID="3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1" exitCode=0 Jul 24 17:11:31.482535 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.482094 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-sgvdq" Jul 24 17:11:31.482535 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.482138 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sgvdq" event={"ID":"aa9640f3-0d6b-4f46-b3ce-21ef384415a8","Type":"ContainerDied","Data":"3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1"} Jul 24 17:11:31.482535 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.482179 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-sgvdq" event={"ID":"aa9640f3-0d6b-4f46-b3ce-21ef384415a8","Type":"ContainerDied","Data":"701901f27c50e36ec967dd0c498b9978f35cd971d7a73a40b45504b130de2a94"} Jul 24 17:11:31.482535 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.482195 2552 scope.go:117] "RemoveContainer" containerID="3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1" Jul 24 17:11:31.491159 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.491129 2552 scope.go:117] "RemoveContainer" containerID="3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1" Jul 24 17:11:31.491453 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:11:31.491432 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1\": container with ID starting with 3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1 not found: ID does not exist" containerID="3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1" Jul 24 17:11:31.491515 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.491464 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1"} err="failed to get container status \"3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1\": rpc error: code = NotFound desc = could not find container \"3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1\": container with ID starting with 3f0e1e07ddf440840eae02e03c8799b821a6d0c2c21eadc99e66f22ce065a4a1 not found: ID does not exist" Jul 24 17:11:31.498538 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.498517 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sgvdq"] Jul 24 17:11:31.505553 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:31.505530 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-sgvdq"] Jul 24 17:11:32.965822 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:32.965789 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9640f3-0d6b-4f46-b3ce-21ef384415a8" path="/var/lib/kubelet/pods/aa9640f3-0d6b-4f46-b3ce-21ef384415a8/volumes" Jul 24 17:11:35.222744 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:35.222688 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:35.344552 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:35.344510 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:35.435828 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:35.435791 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:38.529701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:38.529652 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:38.623101 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:38.623069 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:49.152546 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:49.152504 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:11:49.252998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:11:49.252964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:10.291372 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.291336 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:10.387282 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.387251 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:10.865345 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.865306 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:10.986195 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.986163 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd"] Jul 24 17:12:10.986553 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.986539 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9640f3-0d6b-4f46-b3ce-21ef384415a8" containerName="authorino" Jul 24 17:12:10.986602 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.986555 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9640f3-0d6b-4f46-b3ce-21ef384415a8" containerName="authorino" Jul 24 17:12:10.986638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.986632 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9640f3-0d6b-4f46-b3ce-21ef384415a8" containerName="authorino" Jul 24 17:12:10.990846 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.990830 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:10.994192 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.994171 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 17:12:10.994388 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.994374 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fxv6v\"" Jul 24 17:12:10.994388 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.994382 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 17:12:10.994550 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.994477 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 17:12:10.999376 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:10.999354 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd"] Jul 24 17:12:11.020344 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.020322 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:11.090870 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.090846 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p54cx\" (UniqueName: \"kubernetes.io/projected/92269590-ad86-4c74-bfa9-9bbe70064fee-kube-api-access-p54cx\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.091109 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.091086 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92269590-ad86-4c74-bfa9-9bbe70064fee-maas-api-tls\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.091200 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.091132 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92269590-ad86-4c74-bfa9-9bbe70064fee-openshift-service-ca\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.191718 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.191694 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p54cx\" (UniqueName: \"kubernetes.io/projected/92269590-ad86-4c74-bfa9-9bbe70064fee-kube-api-access-p54cx\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.191827 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.191753 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92269590-ad86-4c74-bfa9-9bbe70064fee-maas-api-tls\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.191827 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.191773 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92269590-ad86-4c74-bfa9-9bbe70064fee-openshift-service-ca\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.192358 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.192337 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92269590-ad86-4c74-bfa9-9bbe70064fee-openshift-service-ca\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.194196 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.194170 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92269590-ad86-4c74-bfa9-9bbe70064fee-maas-api-tls\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.200192 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.200170 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p54cx\" (UniqueName: \"kubernetes.io/projected/92269590-ad86-4c74-bfa9-9bbe70064fee-kube-api-access-p54cx\") pod \"maas-api-5c897c7bdd-8nqkd\" (UID: \"92269590-ad86-4c74-bfa9-9bbe70064fee\") " pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.301515 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.301483 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:11.437666 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.437634 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd"] Jul 24 17:12:11.441833 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:12:11.441771 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92269590_ad86_4c74_bfa9_9bbe70064fee.slice/crio-f5654d3dc59c2730b52f6d3ddae04c9dd037d3412d440e2a70ad2175e3b801db WatchSource:0}: Error finding container f5654d3dc59c2730b52f6d3ddae04c9dd037d3412d440e2a70ad2175e3b801db: Status 404 returned error can't find the container with id f5654d3dc59c2730b52f6d3ddae04c9dd037d3412d440e2a70ad2175e3b801db Jul 24 17:12:11.445790 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.445763 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:11.550246 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.550207 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:11.592063 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.592031 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-95zvq"] Jul 24 17:12:11.596661 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.596642 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:11.599391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.599371 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 17:12:11.599908 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.599892 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-995vp\"" Jul 24 17:12:11.619320 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.619299 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-95zvq"] Jul 24 17:12:11.681420 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.681374 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" event={"ID":"92269590-ad86-4c74-bfa9-9bbe70064fee","Type":"ContainerStarted","Data":"f5654d3dc59c2730b52f6d3ddae04c9dd037d3412d440e2a70ad2175e3b801db"} Jul 24 17:12:11.695663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.695596 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c659f64a-f380-4809-be1e-1f1f77931d8c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-95zvq\" (UID: \"c659f64a-f380-4809-be1e-1f1f77931d8c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:11.695663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.695651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62tn5\" (UniqueName: \"kubernetes.io/projected/c659f64a-f380-4809-be1e-1f1f77931d8c-kube-api-access-62tn5\") pod \"payload-processing-5cc6cd444b-95zvq\" (UID: \"c659f64a-f380-4809-be1e-1f1f77931d8c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:11.796387 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.796353 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62tn5\" (UniqueName: \"kubernetes.io/projected/c659f64a-f380-4809-be1e-1f1f77931d8c-kube-api-access-62tn5\") pod \"payload-processing-5cc6cd444b-95zvq\" (UID: \"c659f64a-f380-4809-be1e-1f1f77931d8c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:11.796593 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.796472 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c659f64a-f380-4809-be1e-1f1f77931d8c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-95zvq\" (UID: \"c659f64a-f380-4809-be1e-1f1f77931d8c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:11.797319 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.797290 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c659f64a-f380-4809-be1e-1f1f77931d8c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-95zvq\" (UID: \"c659f64a-f380-4809-be1e-1f1f77931d8c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:11.809767 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.809746 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62tn5\" (UniqueName: \"kubernetes.io/projected/c659f64a-f380-4809-be1e-1f1f77931d8c-kube-api-access-62tn5\") pod \"payload-processing-5cc6cd444b-95zvq\" (UID: \"c659f64a-f380-4809-be1e-1f1f77931d8c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:11.907608 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:11.907583 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:12.049309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:12.049282 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-95zvq"] Jul 24 17:12:12.051644 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:12:12.051616 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc659f64a_f380_4809_be1e_1f1f77931d8c.slice/crio-b1c65c2c6c881cfa28567c55b1fb9a64948fd0aa6213fd6768ac90fc40bde22d WatchSource:0}: Error finding container b1c65c2c6c881cfa28567c55b1fb9a64948fd0aa6213fd6768ac90fc40bde22d: Status 404 returned error can't find the container with id b1c65c2c6c881cfa28567c55b1fb9a64948fd0aa6213fd6768ac90fc40bde22d Jul 24 17:12:12.690780 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:12.690741 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" event={"ID":"c659f64a-f380-4809-be1e-1f1f77931d8c","Type":"ContainerStarted","Data":"b1c65c2c6c881cfa28567c55b1fb9a64948fd0aa6213fd6768ac90fc40bde22d"} Jul 24 17:12:16.718101 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:16.718057 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" event={"ID":"92269590-ad86-4c74-bfa9-9bbe70064fee","Type":"ContainerStarted","Data":"8b6b2676b24c3e3de893ed7c426e003bd8a6ab195f8aa80e0a7a710b3eebec89"} Jul 24 17:12:16.718580 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:16.718129 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:16.719777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:16.719749 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" event={"ID":"c659f64a-f380-4809-be1e-1f1f77931d8c","Type":"ContainerStarted","Data":"8e4136b9e13541fd5849163485bb0c473a43776012462c7af565fb28d1e7893f"} Jul 24 17:12:16.719880 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:16.719839 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:16.737225 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:16.737174 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" podStartSLOduration=2.298426527 podStartE2EDuration="6.737132746s" totalImagesPullingTime="4.438706219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:12:16.734939062 +0000 UTC m=+554.233341608" watchObservedRunningTime="2026-07-24 17:12:16.737132746 +0000 UTC m=+554.235535291" Jul 24 17:12:16.754604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:16.754541 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" podStartSLOduration=1.862465923 podStartE2EDuration="5.754522189s" totalImagesPullingTime="3.892056266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:12:16.753383826 +0000 UTC m=+554.251786373" watchObservedRunningTime="2026-07-24 17:12:16.754522189 +0000 UTC m=+554.252924735" Jul 24 17:12:19.736340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:19.736306 2552 generic.go:356] "Generic (PLEG): container finished" podID="78961fb3-3403-4f44-a263-0e730fdb851a" containerID="6ee352c617bb303955160fa7c5bd22eef4f7bf377fb499212d8465ed766c5b81" exitCode=0 Jul 24 17:12:19.736787 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:19.736379 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" event={"ID":"78961fb3-3403-4f44-a263-0e730fdb851a","Type":"ContainerDied","Data":"6ee352c617bb303955160fa7c5bd22eef4f7bf377fb499212d8465ed766c5b81"} Jul 24 17:12:19.736787 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:19.736780 2552 scope.go:117] "RemoveContainer" containerID="6ee352c617bb303955160fa7c5bd22eef4f7bf377fb499212d8465ed766c5b81" Jul 24 17:12:19.737709 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:19.737688 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:12:19.737767 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:19.737750 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:12:19.737803 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:19.737783 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:12:20.747097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:20.747050 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" event={"ID":"78961fb3-3403-4f44-a263-0e730fdb851a","Type":"ContainerStarted","Data":"435ab4cc544783c954fe8f9541df4a213ddb0829b6e5bd894b57d3bb47d78751"} Jul 24 17:12:20.874709 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:20.874670 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:12:20.876022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:20.875994 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpt5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 17:12:20.876139 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:20.876050 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" podUID="78961fb3-3403-4f44-a263-0e730fdb851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 17:12:21.875721 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:21.875677 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpt5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 17:12:21.876229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:21.875747 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" podUID="78961fb3-3403-4f44-a263-0e730fdb851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 17:12:22.738826 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:22.738783 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c897c7bdd-8nqkd" Jul 24 17:12:22.875068 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:22.875027 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpt5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 17:12:22.875290 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:22.875090 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" podUID="78961fb3-3403-4f44-a263-0e730fdb851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 17:12:23.726791 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:23.726751 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:23.827430 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:23.827382 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:23.874818 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:23.874783 2552 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpt5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 17:12:23.875005 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:23.874838 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" podUID="78961fb3-3403-4f44-a263-0e730fdb851a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 17:12:24.878991 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:24.878963 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:12:24.879425 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:24.879251 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:12:24.880011 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:24.879995 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpt5t" Jul 24 17:12:27.727963 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:27.727929 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-95zvq" Jul 24 17:12:29.310982 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:29.310940 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:42.551475 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:42.551437 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:45.690922 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:45.690884 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:45.837553 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:45.837511 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:12:50.101372 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:12:50.101339 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:13:02.909443 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:02.909393 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:13:02.909879 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:02.909493 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:13:03.703387 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.703352 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx"] Jul 24 17:13:03.706929 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.706913 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.710508 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.710486 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 17:13:03.712608 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.712580 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 17:13:03.713232 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.713213 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-85ddw\"" Jul 24 17:13:03.713781 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.713764 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 17:13:03.727260 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.727228 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:13:03.740871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.740844 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx"] Jul 24 17:13:03.793461 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.793393 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.793461 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.793455 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbss\" (UniqueName: \"kubernetes.io/projected/873a1359-6830-4840-a971-6b100082dac1-kube-api-access-dcbss\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.793698 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.793513 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.793698 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.793563 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.793698 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.793582 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873a1359-6830-4840-a971-6b100082dac1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.793698 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.793664 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894193 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894146 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894193 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894190 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873a1359-6830-4840-a971-6b100082dac1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894499 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894225 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894499 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894295 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894499 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894321 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbss\" (UniqueName: \"kubernetes.io/projected/873a1359-6830-4840-a971-6b100082dac1-kube-api-access-dcbss\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894499 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894349 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894717 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894642 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894717 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894688 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.894788 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.894724 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.896692 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.896674 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/873a1359-6830-4840-a971-6b100082dac1-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.897005 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.896984 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/873a1359-6830-4840-a971-6b100082dac1-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:03.907290 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:03.907262 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbss\" (UniqueName: \"kubernetes.io/projected/873a1359-6830-4840-a971-6b100082dac1-kube-api-access-dcbss\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx\" (UID: \"873a1359-6830-4840-a971-6b100082dac1\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:04.018165 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:04.018044 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:04.164775 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:04.164745 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx"] Jul 24 17:13:04.166154 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:13:04.166121 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873a1359_6830_4840_a971_6b100082dac1.slice/crio-b4def8706c64b9238c7ada14f392763efaef686d9ef3c573edf8b19921fd3975 WatchSource:0}: Error finding container b4def8706c64b9238c7ada14f392763efaef686d9ef3c573edf8b19921fd3975: Status 404 returned error can't find the container with id b4def8706c64b9238c7ada14f392763efaef686d9ef3c573edf8b19921fd3975 Jul 24 17:13:04.973421 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:04.973371 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" event={"ID":"873a1359-6830-4840-a971-6b100082dac1","Type":"ContainerStarted","Data":"b4def8706c64b9238c7ada14f392763efaef686d9ef3c573edf8b19921fd3975"} Jul 24 17:13:05.980050 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:05.980016 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" event={"ID":"873a1359-6830-4840-a971-6b100082dac1","Type":"ContainerStarted","Data":"afdda2a8c4c28d27db97386e29af102d540a5d857d2417bdca4e7084a63083bd"} Jul 24 17:13:06.018949 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:06.018892 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" podStartSLOduration=1.935617122 podStartE2EDuration="3.018875933s" totalImagesPullingTime="1.083258811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:13:06.018797219 +0000 UTC m=+603.517199765" watchObservedRunningTime="2026-07-24 17:13:06.018875933 +0000 UTC m=+603.517278479" Jul 24 17:13:07.073916 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:07.073881 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:13:10.108273 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:10.108210 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:13:14.019045 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:14.019006 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:14.019573 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:14.019084 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:14.032212 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:14.032183 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:15.037564 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:15.037535 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx" Jul 24 17:13:20.334442 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:20.334387 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:13:21.583689 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:21.583647 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:13:51.381421 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:13:51.381375 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:14:01.208548 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:01.208503 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:14:27.731641 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.731605 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59b78ff696-gd8c7"] Jul 24 17:14:27.735682 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.735663 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:27.738266 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.738241 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vxxvf\"" Jul 24 17:14:27.739100 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.739084 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 17:14:27.746569 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.746545 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59b78ff696-gd8c7"] Jul 24 17:14:27.867015 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.866976 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62bce08a-c14d-469c-9605-c9aee71fcf3b-tls-cert\") pod \"authorino-59b78ff696-gd8c7\" (UID: \"62bce08a-c14d-469c-9605-c9aee71fcf3b\") " pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:27.867193 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.867031 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdpm\" (UniqueName: \"kubernetes.io/projected/62bce08a-c14d-469c-9605-c9aee71fcf3b-kube-api-access-5qdpm\") pod \"authorino-59b78ff696-gd8c7\" (UID: \"62bce08a-c14d-469c-9605-c9aee71fcf3b\") " pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:27.967867 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.967830 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62bce08a-c14d-469c-9605-c9aee71fcf3b-tls-cert\") pod \"authorino-59b78ff696-gd8c7\" (UID: \"62bce08a-c14d-469c-9605-c9aee71fcf3b\") " pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:27.968038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.967878 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdpm\" (UniqueName: \"kubernetes.io/projected/62bce08a-c14d-469c-9605-c9aee71fcf3b-kube-api-access-5qdpm\") pod \"authorino-59b78ff696-gd8c7\" (UID: \"62bce08a-c14d-469c-9605-c9aee71fcf3b\") " pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:27.970171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.970142 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62bce08a-c14d-469c-9605-c9aee71fcf3b-tls-cert\") pod \"authorino-59b78ff696-gd8c7\" (UID: \"62bce08a-c14d-469c-9605-c9aee71fcf3b\") " pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:27.978102 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:27.978079 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdpm\" (UniqueName: \"kubernetes.io/projected/62bce08a-c14d-469c-9605-c9aee71fcf3b-kube-api-access-5qdpm\") pod \"authorino-59b78ff696-gd8c7\" (UID: \"62bce08a-c14d-469c-9605-c9aee71fcf3b\") " pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:28.045592 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:28.045508 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59b78ff696-gd8c7" Jul 24 17:14:28.173029 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:28.173000 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:14:28.199658 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:28.199631 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59b78ff696-gd8c7"] Jul 24 17:14:28.201585 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:14:28.201553 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62bce08a_c14d_469c_9605_c9aee71fcf3b.slice/crio-7ffdce2b8adaf977a694246e4d2c490dcc36301acafa623e3726855f0db37a19 WatchSource:0}: Error finding container 7ffdce2b8adaf977a694246e4d2c490dcc36301acafa623e3726855f0db37a19: Status 404 returned error can't find the container with id 7ffdce2b8adaf977a694246e4d2c490dcc36301acafa623e3726855f0db37a19 Jul 24 17:14:28.202903 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:28.202883 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:14:28.342610 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:28.342506 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:14:28.393910 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:28.393878 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59b78ff696-gd8c7" event={"ID":"62bce08a-c14d-469c-9605-c9aee71fcf3b","Type":"ContainerStarted","Data":"7ffdce2b8adaf977a694246e4d2c490dcc36301acafa623e3726855f0db37a19"} Jul 24 17:14:29.396903 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:29.396863 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:14:29.403269 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:29.403228 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59b78ff696-gd8c7" event={"ID":"62bce08a-c14d-469c-9605-c9aee71fcf3b","Type":"ContainerStarted","Data":"75ab3ddf66c34e6713cd76c1bd113bf943c669f456ff8c491a577f74c0380ade"} Jul 24 17:14:29.474460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:29.474371 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59b78ff696-gd8c7" podStartSLOduration=2.19726084 podStartE2EDuration="2.474350816s" totalImagesPullingTime="277.089976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:14:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:14:29.472090413 +0000 UTC m=+686.970492959" watchObservedRunningTime="2026-07-24 17:14:29.474350816 +0000 UTC m=+686.972753366" Jul 24 17:14:29.548717 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:14:29.548681 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:15:29.803530 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:15:29.803483 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:15:34.143546 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:15:34.143507 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:15:34.201416 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:15:34.201381 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:16:02.968279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:16:02.968250 2552 scope.go:117] "RemoveContainer" containerID="20d3a76944af5407e0dae5c3242233fbd507b4052ea3feb2b3292442dc29b554" Jul 24 17:16:31.905377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:16:31.905346 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:16:34.335900 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:16:34.335858 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:16:35.530840 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:16:35.530800 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:11.286687 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:11.286646 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:11.388113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:11.388063 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:11.521148 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:11.521120 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:17.285158 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:17.285114 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:17.391933 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:17.391883 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:17.543900 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:17.543811 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:17.672316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:17.672283 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:23.440687 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:23.440644 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:17:23.544973 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:17:23.544933 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:18:02.940022 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:18:02.939939 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:18:02.944020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:18:02.943997 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:20:12.502773 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:20:12.502730 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:21:24.486812 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:24.486777 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf"] Jul 24 17:21:24.487301 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:24.486971 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" podUID="c66050e2-11ab-4070-aa52-9ad5aa4896d2" containerName="manager" containerID="cri-o://330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac" gracePeriod=10 Jul 24 17:21:28.143663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.143636 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:21:28.236859 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.236770 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66050e2-11ab-4070-aa52-9ad5aa4896d2-extensions-socket-volume\") pod \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " Jul 24 17:21:28.236859 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.236858 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf6jt\" (UniqueName: \"kubernetes.io/projected/c66050e2-11ab-4070-aa52-9ad5aa4896d2-kube-api-access-xf6jt\") pod \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\" (UID: \"c66050e2-11ab-4070-aa52-9ad5aa4896d2\") " Jul 24 17:21:28.237167 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.237143 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c66050e2-11ab-4070-aa52-9ad5aa4896d2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c66050e2-11ab-4070-aa52-9ad5aa4896d2" (UID: "c66050e2-11ab-4070-aa52-9ad5aa4896d2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:21:28.238859 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.238839 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66050e2-11ab-4070-aa52-9ad5aa4896d2-kube-api-access-xf6jt" (OuterVolumeSpecName: "kube-api-access-xf6jt") pod "c66050e2-11ab-4070-aa52-9ad5aa4896d2" (UID: "c66050e2-11ab-4070-aa52-9ad5aa4896d2"). InnerVolumeSpecName "kube-api-access-xf6jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:21:28.337854 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.337814 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf6jt\" (UniqueName: \"kubernetes.io/projected/c66050e2-11ab-4070-aa52-9ad5aa4896d2-kube-api-access-xf6jt\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:21:28.337854 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.337845 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c66050e2-11ab-4070-aa52-9ad5aa4896d2-extensions-socket-volume\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:21:28.564927 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.564835 2552 generic.go:356] "Generic (PLEG): container finished" podID="c66050e2-11ab-4070-aa52-9ad5aa4896d2" containerID="330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac" exitCode=0 Jul 24 17:21:28.564927 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.564879 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" Jul 24 17:21:28.565113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.564876 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" event={"ID":"c66050e2-11ab-4070-aa52-9ad5aa4896d2","Type":"ContainerDied","Data":"330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac"} Jul 24 17:21:28.565113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.564991 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf" event={"ID":"c66050e2-11ab-4070-aa52-9ad5aa4896d2","Type":"ContainerDied","Data":"e9c2b53592eb3cf18faaf5bd6e383eee2d6bfed0ae3e8fbfe8b9ccf480355484"} Jul 24 17:21:28.565113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.565016 2552 scope.go:117] "RemoveContainer" containerID="330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac" Jul 24 17:21:28.575235 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.575216 2552 scope.go:117] "RemoveContainer" containerID="330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac" Jul 24 17:21:28.575507 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:21:28.575488 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac\": container with ID starting with 330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac not found: ID does not exist" containerID="330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac" Jul 24 17:21:28.575559 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.575517 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac"} err="failed to get container status \"330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac\": rpc error: code = NotFound desc = could not find container \"330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac\": container with ID starting with 330ea40bf46ac29b6e6423a46f72abb0811d7a2c06b5a6323a151ffe1bbd50ac not found: ID does not exist" Jul 24 17:21:28.604930 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.604898 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf"] Jul 24 17:21:28.610090 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.610064 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hs6sf"] Jul 24 17:21:28.964487 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:28.964457 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66050e2-11ab-4070-aa52-9ad5aa4896d2" path="/var/lib/kubelet/pods/c66050e2-11ab-4070-aa52-9ad5aa4896d2/volumes" Jul 24 17:21:50.924149 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:21:50.924113 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:22:25.615811 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.615726 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688"] Jul 24 17:22:25.618229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.616189 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66050e2-11ab-4070-aa52-9ad5aa4896d2" containerName="manager" Jul 24 17:22:25.618229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.616203 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66050e2-11ab-4070-aa52-9ad5aa4896d2" containerName="manager" Jul 24 17:22:25.618229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.616264 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66050e2-11ab-4070-aa52-9ad5aa4896d2" containerName="manager" Jul 24 17:22:25.619345 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.619331 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:25.621969 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.621947 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wjdgd\"" Jul 24 17:22:25.631949 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.631928 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688"] Jul 24 17:22:25.748007 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.747975 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbl4n\" (UniqueName: \"kubernetes.io/projected/06763651-dbb9-4379-a734-830ad94ac506-kube-api-access-jbl4n\") pod \"kuadrant-operator-controller-manager-86878c769f-gc688\" (UID: \"06763651-dbb9-4379-a734-830ad94ac506\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:25.748181 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.748071 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06763651-dbb9-4379-a734-830ad94ac506-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gc688\" (UID: \"06763651-dbb9-4379-a734-830ad94ac506\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:25.849382 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.849355 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06763651-dbb9-4379-a734-830ad94ac506-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gc688\" (UID: \"06763651-dbb9-4379-a734-830ad94ac506\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:25.849550 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.849444 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbl4n\" (UniqueName: \"kubernetes.io/projected/06763651-dbb9-4379-a734-830ad94ac506-kube-api-access-jbl4n\") pod \"kuadrant-operator-controller-manager-86878c769f-gc688\" (UID: \"06763651-dbb9-4379-a734-830ad94ac506\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:25.849823 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.849800 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/06763651-dbb9-4379-a734-830ad94ac506-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gc688\" (UID: \"06763651-dbb9-4379-a734-830ad94ac506\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:25.868436 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.868366 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbl4n\" (UniqueName: \"kubernetes.io/projected/06763651-dbb9-4379-a734-830ad94ac506-kube-api-access-jbl4n\") pod \"kuadrant-operator-controller-manager-86878c769f-gc688\" (UID: \"06763651-dbb9-4379-a734-830ad94ac506\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:25.929057 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:25.929030 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:26.069286 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:26.069260 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688"] Jul 24 17:22:26.070784 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:22:26.070753 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06763651_dbb9_4379_a734_830ad94ac506.slice/crio-4303f90ab2467579a6fc8d1ddc407b28f142fd1b62f94b81c9b429fce4e2117a WatchSource:0}: Error finding container 4303f90ab2467579a6fc8d1ddc407b28f142fd1b62f94b81c9b429fce4e2117a: Status 404 returned error can't find the container with id 4303f90ab2467579a6fc8d1ddc407b28f142fd1b62f94b81c9b429fce4e2117a Jul 24 17:22:26.850255 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:26.850219 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" event={"ID":"06763651-dbb9-4379-a734-830ad94ac506","Type":"ContainerStarted","Data":"57472a49284ba08be83e95868667a13a530e0e552e9b2d73d72db19ed6f50b03"} Jul 24 17:22:26.850696 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:26.850300 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" event={"ID":"06763651-dbb9-4379-a734-830ad94ac506","Type":"ContainerStarted","Data":"4303f90ab2467579a6fc8d1ddc407b28f142fd1b62f94b81c9b429fce4e2117a"} Jul 24 17:22:26.850696 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:26.850384 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:22:26.875001 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:26.874956 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" podStartSLOduration=1.874942527 podStartE2EDuration="1.874942527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:22:26.872018764 +0000 UTC m=+1164.370421327" watchObservedRunningTime="2026-07-24 17:22:26.874942527 +0000 UTC m=+1164.373345072" Jul 24 17:22:37.857676 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:22:37.857644 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gc688" Jul 24 17:23:02.981823 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:23:02.981792 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:23:02.987371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:23:02.987352 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:25:10.546808 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:25:10.546778 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:25:28.209994 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:25:28.209961 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:25:36.907208 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:25:36.907175 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:25:43.956432 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:25:43.956380 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:25:51.340941 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:25:51.340890 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:25:56.097889 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:25:56.097857 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:25:57.653225 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:25:57.653165 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:28:03.011004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:28:03.010977 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:28:03.021516 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:28:03.021496 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:29:05.209145 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:05.209108 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:23.617066 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:23.617033 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:26.687679 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:26.687599 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:26.785788 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:26.785754 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:27.276643 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:27.276600 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:33.041043 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.041009 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:33.141899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.141864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:33.334233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.334147 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:33.691197 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.691164 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4"] Jul 24 17:29:33.694636 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.694619 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.697179 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.697155 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6432af27-openshift-default-dockercfg-56bwp\"" Jul 24 17:29:33.720708 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.720209 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4"] Jul 24 17:29:33.723821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.723787 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:33.814715 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814680 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-token\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.814926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814728 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-credential-socket\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.814926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814823 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-data\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.814926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814859 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-envoy\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.814926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814889 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-socket\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.815097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814947 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-podinfo\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.815097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814963 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-certs\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.815097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.814994 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istiod-ca-cert\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.815097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.815015 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q48p5\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-kube-api-access-q48p5\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.915865 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.915827 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-credential-socket\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.915865 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.915870 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-data\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916103 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.915906 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-envoy\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916103 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.915925 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-socket\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916103 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.915964 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-podinfo\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916103 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.915986 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-certs\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916103 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916022 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istiod-ca-cert\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916103 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916049 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q48p5\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-kube-api-access-q48p5\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916364 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916105 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-token\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916364 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916249 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-credential-socket\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916364 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916311 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-data\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916587 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916566 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-socket\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916622 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-certs\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.916755 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.916737 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istiod-ca-cert\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.918640 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.918612 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-envoy\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.918807 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.918783 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-podinfo\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.925139 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.925114 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q48p5\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-kube-api-access-q48p5\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.925363 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.925346 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-token\") pod \"e2e-ait-6432af27-openshift-default-97d86486-jpzw4\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:33.945562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:33.945486 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:34.005336 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:34.005301 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:34.152180 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:34.152156 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4"] Jul 24 17:29:34.154228 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:29:34.154189 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f1ae3e_4725_487e_86da_5e0b38194e6b.slice/crio-875deda8977b8f8e0d60858ce4499b1f2ee1a8540a656de1d8cd596ab84e213d WatchSource:0}: Error finding container 875deda8977b8f8e0d60858ce4499b1f2ee1a8540a656de1d8cd596ab84e213d: Status 404 returned error can't find the container with id 875deda8977b8f8e0d60858ce4499b1f2ee1a8540a656de1d8cd596ab84e213d Jul 24 17:29:34.156385 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:34.156353 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:29:34.156482 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:34.156469 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:29:34.156532 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:34.156509 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:29:35.037920 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:35.037884 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" event={"ID":"b6f1ae3e-4725-487e-86da-5e0b38194e6b","Type":"ContainerStarted","Data":"3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2"} Jul 24 17:29:35.038104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:35.037940 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" event={"ID":"b6f1ae3e-4725-487e-86da-5e0b38194e6b","Type":"ContainerStarted","Data":"875deda8977b8f8e0d60858ce4499b1f2ee1a8540a656de1d8cd596ab84e213d"} Jul 24 17:29:35.061281 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:35.061228 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" podStartSLOduration=2.061212667 podStartE2EDuration="2.061212667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:35.059660393 +0000 UTC m=+1592.558062963" watchObservedRunningTime="2026-07-24 17:29:35.061212667 +0000 UTC m=+1592.559615210" Jul 24 17:29:36.005913 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.005875 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:36.010668 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.010644 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:36.042340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.042302 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:36.043302 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.043285 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:36.422108 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.422071 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:36.509712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.509679 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:36.627675 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.627644 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46"] Jul 24 17:29:36.631304 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.631281 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.634294 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.634260 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-ait-6432af27\"" Jul 24 17:29:36.646988 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.646961 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46"] Jul 24 17:29:36.691795 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.691706 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:36.743983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.743953 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beceb10e-1720-4177-b3e3-f25ca98093ae-openshift-service-ca\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.744169 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.744014 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.744169 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.744110 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rpl\" (UniqueName: \"kubernetes.io/projected/beceb10e-1720-4177-b3e3-f25ca98093ae-kube-api-access-q6rpl\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.845228 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.845181 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beceb10e-1720-4177-b3e3-f25ca98093ae-openshift-service-ca\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.845462 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.845278 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.845462 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.845320 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rpl\" (UniqueName: \"kubernetes.io/projected/beceb10e-1720-4177-b3e3-f25ca98093ae-kube-api-access-q6rpl\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.845608 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:29:36.845455 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-ait-6432af27: secret "maas-api-serving-cert-e2e-ait-6432af27" not found Jul 24 17:29:36.845608 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:29:36.845544 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls podName:beceb10e-1720-4177-b3e3-f25ca98093ae nodeName:}" failed. No retries permitted until 2026-07-24 17:29:37.34552286 +0000 UTC m=+1594.843925407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls") pod "maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" (UID: "beceb10e-1720-4177-b3e3-f25ca98093ae") : secret "maas-api-serving-cert-e2e-ait-6432af27" not found Jul 24 17:29:36.845957 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.845939 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beceb10e-1720-4177-b3e3-f25ca98093ae-openshift-service-ca\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.857136 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.857112 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rpl\" (UniqueName: \"kubernetes.io/projected/beceb10e-1720-4177-b3e3-f25ca98093ae-kube-api-access-q6rpl\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:36.963857 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:36.963767 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:37.214873 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:37.214787 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:37.317871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:37.317839 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:29:37.350148 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:37.350109 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:37.352517 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:37.352493 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls\") pod \"maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:37.544897 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:37.544806 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:37.689645 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:37.689585 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46"] Jul 24 17:29:37.691975 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:29:37.691946 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeceb10e_1720_4177_b3e3_f25ca98093ae.slice/crio-a0ca4071fdc14ba2934a24624a028a5015f9f0a7a661bcf1bd67295cdfc56bca WatchSource:0}: Error finding container a0ca4071fdc14ba2934a24624a028a5015f9f0a7a661bcf1bd67295cdfc56bca: Status 404 returned error can't find the container with id a0ca4071fdc14ba2934a24624a028a5015f9f0a7a661bcf1bd67295cdfc56bca Jul 24 17:29:37.693464 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:37.693447 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:29:38.052655 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:38.052623 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" event={"ID":"beceb10e-1720-4177-b3e3-f25ca98093ae","Type":"ContainerStarted","Data":"a0ca4071fdc14ba2934a24624a028a5015f9f0a7a661bcf1bd67295cdfc56bca"} Jul 24 17:29:39.059488 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:39.059448 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" event={"ID":"beceb10e-1720-4177-b3e3-f25ca98093ae","Type":"ContainerStarted","Data":"21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf"} Jul 24 17:29:39.059891 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:39.059551 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:39.118509 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:39.118401 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" podStartSLOduration=2.584000243 podStartE2EDuration="3.118378973s" totalImagesPullingTime="534.37873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:29:39.117324001 +0000 UTC m=+1596.615726548" watchObservedRunningTime="2026-07-24 17:29:39.118378973 +0000 UTC m=+1596.616781520" Jul 24 17:29:45.071857 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:45.071828 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:29:52.083620 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:52.083571 2552 patch_prober.go:28] interesting pod/e2e-ait-6432af27-openshift-default-97d86486-jpzw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 17:29:52.084139 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:52.083648 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 17:29:53.134752 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:53.134721 2552 generic.go:356] "Generic (PLEG): container finished" podID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerID="3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2" exitCode=0 Jul 24 17:29:53.135180 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:53.134805 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" event={"ID":"b6f1ae3e-4725-487e-86da-5e0b38194e6b","Type":"ContainerDied","Data":"3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2"} Jul 24 17:29:53.135252 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:53.135235 2552 scope.go:117] "RemoveContainer" containerID="3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2" Jul 24 17:29:53.143008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:53.136838 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:29:53.143008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:53.137018 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:29:53.143008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:53.137115 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:29:54.145518 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:54.145485 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" event={"ID":"b6f1ae3e-4725-487e-86da-5e0b38194e6b","Type":"ContainerStarted","Data":"95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88"} Jul 24 17:29:55.005729 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:55.005686 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:55.005976 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:55.005951 2552 patch_prober.go:28] interesting pod/e2e-ait-6432af27-openshift-default-97d86486-jpzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 17:29:55.006063 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:55.006011 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 17:29:56.006046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:56.006007 2552 patch_prober.go:28] interesting pod/e2e-ait-6432af27-openshift-default-97d86486-jpzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 17:29:56.006453 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:56.006067 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 17:29:57.006531 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:57.006498 2552 patch_prober.go:28] interesting pod/e2e-ait-6432af27-openshift-default-97d86486-jpzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 17:29:57.006909 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:57.006551 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 17:29:58.005688 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:58.005645 2552 patch_prober.go:28] interesting pod/e2e-ait-6432af27-openshift-default-97d86486-jpzw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 24 17:29:58.005921 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:58.005723 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 24 17:29:59.009977 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:59.009947 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:59.010356 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:59.010227 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:29:59.010983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:29:59.010967 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:30:12.418873 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.418841 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46"] Jul 24 17:30:12.419686 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.419086 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" podUID="beceb10e-1720-4177-b3e3-f25ca98093ae" containerName="maas-api" containerID="cri-o://21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf" gracePeriod=30 Jul 24 17:30:12.445489 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.445461 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:12.555038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.554999 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:12.654046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.654008 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:12.682904 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.682821 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:30:12.790518 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.790488 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beceb10e-1720-4177-b3e3-f25ca98093ae-openshift-service-ca\") pod \"beceb10e-1720-4177-b3e3-f25ca98093ae\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " Jul 24 17:30:12.790518 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.790520 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls\") pod \"beceb10e-1720-4177-b3e3-f25ca98093ae\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " Jul 24 17:30:12.790771 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.790549 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6rpl\" (UniqueName: \"kubernetes.io/projected/beceb10e-1720-4177-b3e3-f25ca98093ae-kube-api-access-q6rpl\") pod \"beceb10e-1720-4177-b3e3-f25ca98093ae\" (UID: \"beceb10e-1720-4177-b3e3-f25ca98093ae\") " Jul 24 17:30:12.790988 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.790962 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beceb10e-1720-4177-b3e3-f25ca98093ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "beceb10e-1720-4177-b3e3-f25ca98093ae" (UID: "beceb10e-1720-4177-b3e3-f25ca98093ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:30:12.792715 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.792692 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beceb10e-1720-4177-b3e3-f25ca98093ae-kube-api-access-q6rpl" (OuterVolumeSpecName: "kube-api-access-q6rpl") pod "beceb10e-1720-4177-b3e3-f25ca98093ae" (UID: "beceb10e-1720-4177-b3e3-f25ca98093ae"). InnerVolumeSpecName "kube-api-access-q6rpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:30:12.792814 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.792716 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "beceb10e-1720-4177-b3e3-f25ca98093ae" (UID: "beceb10e-1720-4177-b3e3-f25ca98093ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:30:12.891889 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.891856 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beceb10e-1720-4177-b3e3-f25ca98093ae-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:12.891889 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.891889 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beceb10e-1720-4177-b3e3-f25ca98093ae-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:12.892126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:12.891901 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6rpl\" (UniqueName: \"kubernetes.io/projected/beceb10e-1720-4177-b3e3-f25ca98093ae-kube-api-access-q6rpl\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:13.242659 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.242571 2552 generic.go:356] "Generic (PLEG): container finished" podID="beceb10e-1720-4177-b3e3-f25ca98093ae" containerID="21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf" exitCode=0 Jul 24 17:30:13.242659 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.242620 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" Jul 24 17:30:13.242910 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.242666 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" event={"ID":"beceb10e-1720-4177-b3e3-f25ca98093ae","Type":"ContainerDied","Data":"21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf"} Jul 24 17:30:13.242910 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.242727 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46" event={"ID":"beceb10e-1720-4177-b3e3-f25ca98093ae","Type":"ContainerDied","Data":"a0ca4071fdc14ba2934a24624a028a5015f9f0a7a661bcf1bd67295cdfc56bca"} Jul 24 17:30:13.242910 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.242757 2552 scope.go:117] "RemoveContainer" containerID="21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf" Jul 24 17:30:13.251696 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.251675 2552 scope.go:117] "RemoveContainer" containerID="21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf" Jul 24 17:30:13.251982 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:30:13.251966 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf\": container with ID starting with 21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf not found: ID does not exist" containerID="21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf" Jul 24 17:30:13.252041 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.251989 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf"} err="failed to get container status \"21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf\": rpc error: code = NotFound desc = could not find container \"21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf\": container with ID starting with 21d63b1714e9f8c0b27510b62065ea82608ef4505fbb1c257a8092c4805a64bf not found: ID does not exist" Jul 24 17:30:13.259362 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.259333 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46"] Jul 24 17:30:13.263064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:13.263040 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6432af27-65bfcf57f6-tvx46"] Jul 24 17:30:14.965083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:14.965048 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beceb10e-1720-4177-b3e3-f25ca98093ae" path="/var/lib/kubelet/pods/beceb10e-1720-4177-b3e3-f25ca98093ae/volumes" Jul 24 17:30:17.072771 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.072740 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:17.281826 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.281791 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4"] Jul 24 17:30:17.282020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.281992 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" containerID="cri-o://95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88" gracePeriod=30 Jul 24 17:30:17.380847 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.380766 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:17.528032 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.527980 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:17.769023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.768982 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:17.868373 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.868340 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:17.968237 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:17.968207 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:18.291197 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:18.291157 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:18.593537 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:18.593393 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:22.546759 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.546702 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:30:22.680675 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.680647 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-token\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.680675 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.680684 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-data\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.680915 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.680707 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-envoy\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.680915 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.680771 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-certs\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.680915 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.680812 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istiod-ca-cert\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.680915 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.680862 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-socket\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.681127 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681006 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q48p5\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-kube-api-access-q48p5\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.681127 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681093 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-credential-socket\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.681244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681133 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-podinfo\") pod \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\" (UID: \"b6f1ae3e-4725-487e-86da-5e0b38194e6b\") " Jul 24 17:30:22.681244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681141 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:30:22.681244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681194 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:30:22.681393 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681364 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:30:22.681495 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681476 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-certs\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.681556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681529 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-workload-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.681556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681545 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-credential-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.681671 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681614 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:30:22.681708 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.681670 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:30:22.683089 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.683061 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:30:22.683191 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.683132 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:30:22.683247 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.683193 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:30:22.683447 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.683401 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-kube-api-access-q48p5" (OuterVolumeSpecName: "kube-api-access-q48p5") pod "b6f1ae3e-4725-487e-86da-5e0b38194e6b" (UID: "b6f1ae3e-4725-487e-86da-5e0b38194e6b"). InnerVolumeSpecName "kube-api-access-q48p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:30:22.782140 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.782107 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-token\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.782140 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.782134 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-data\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.782140 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.782142 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-envoy\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.782140 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.782150 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istiod-ca-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.782431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.782161 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q48p5\" (UniqueName: \"kubernetes.io/projected/b6f1ae3e-4725-487e-86da-5e0b38194e6b-kube-api-access-q48p5\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:22.782431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:22.782170 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6f1ae3e-4725-487e-86da-5e0b38194e6b-istio-podinfo\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:30:23.293777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.293688 2552 generic.go:356] "Generic (PLEG): container finished" podID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerID="95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88" exitCode=0 Jul 24 17:30:23.293777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.293750 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" Jul 24 17:30:23.293777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.293762 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" event={"ID":"b6f1ae3e-4725-487e-86da-5e0b38194e6b","Type":"ContainerDied","Data":"95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88"} Jul 24 17:30:23.293978 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.293811 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4" event={"ID":"b6f1ae3e-4725-487e-86da-5e0b38194e6b","Type":"ContainerDied","Data":"875deda8977b8f8e0d60858ce4499b1f2ee1a8540a656de1d8cd596ab84e213d"} Jul 24 17:30:23.293978 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.293835 2552 scope.go:117] "RemoveContainer" containerID="95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88" Jul 24 17:30:23.302651 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.302634 2552 scope.go:117] "RemoveContainer" containerID="3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2" Jul 24 17:30:23.310913 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.310894 2552 scope.go:117] "RemoveContainer" containerID="95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88" Jul 24 17:30:23.311185 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:30:23.311164 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88\": container with ID starting with 95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88 not found: ID does not exist" containerID="95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88" Jul 24 17:30:23.311251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.311194 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88"} err="failed to get container status \"95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88\": rpc error: code = NotFound desc = could not find container \"95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88\": container with ID starting with 95aa0d5afebc1d5a05664b49e3768f527c2d70a81c13e9ec2ed4d7c0a128ef88 not found: ID does not exist" Jul 24 17:30:23.311251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.311214 2552 scope.go:117] "RemoveContainer" containerID="3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2" Jul 24 17:30:23.311474 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:30:23.311458 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2\": container with ID starting with 3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2 not found: ID does not exist" containerID="3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2" Jul 24 17:30:23.311542 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.311476 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2"} err="failed to get container status \"3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2\": rpc error: code = NotFound desc = could not find container \"3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2\": container with ID starting with 3eed054956afca1f8235d2326f8c60a866037c3f9a2b7875c963c41bb9e33ea2 not found: ID does not exist" Jul 24 17:30:23.311989 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.311969 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4"] Jul 24 17:30:23.315584 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:23.315562 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6432af27-openshift-default-97d86486-jpzw4"] Jul 24 17:30:24.965504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:24.965465 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" path="/var/lib/kubelet/pods/b6f1ae3e-4725-487e-86da-5e0b38194e6b/volumes" Jul 24 17:30:27.183934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.183894 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:27.262466 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262430 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl"] Jul 24 17:30:27.262824 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262810 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beceb10e-1720-4177-b3e3-f25ca98093ae" containerName="maas-api" Jul 24 17:30:27.262824 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262825 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="beceb10e-1720-4177-b3e3-f25ca98093ae" containerName="maas-api" Jul 24 17:30:27.262915 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262836 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" Jul 24 17:30:27.262915 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262844 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" Jul 24 17:30:27.262988 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262930 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" Jul 24 17:30:27.262988 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262941 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" Jul 24 17:30:27.262988 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.262948 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="beceb10e-1720-4177-b3e3-f25ca98093ae" containerName="maas-api" Jul 24 17:30:27.267752 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.267728 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.273618 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.273597 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-migrate-1f7abc01\"" Jul 24 17:30:27.282309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.282286 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl"] Jul 24 17:30:27.297570 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.297543 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:27.422620 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.422534 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5befb0-2d92-49ef-8679-eb19b9cbae19-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.422620 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.422590 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5befb0-2d92-49ef-8679-eb19b9cbae19-maas-api-tls\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.422871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.422748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj2b4\" (UniqueName: \"kubernetes.io/projected/de5befb0-2d92-49ef-8679-eb19b9cbae19-kube-api-access-fj2b4\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.501546 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.501464 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:27.523335 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.523304 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj2b4\" (UniqueName: \"kubernetes.io/projected/de5befb0-2d92-49ef-8679-eb19b9cbae19-kube-api-access-fj2b4\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.523527 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.523354 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5befb0-2d92-49ef-8679-eb19b9cbae19-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.523527 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.523394 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5befb0-2d92-49ef-8679-eb19b9cbae19-maas-api-tls\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.524035 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.524016 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5befb0-2d92-49ef-8679-eb19b9cbae19-openshift-service-ca\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.525752 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.525729 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5befb0-2d92-49ef-8679-eb19b9cbae19-maas-api-tls\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.531250 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.531225 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj2b4\" (UniqueName: \"kubernetes.io/projected/de5befb0-2d92-49ef-8679-eb19b9cbae19-kube-api-access-fj2b4\") pod \"maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.578281 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.578250 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:27.742748 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.742577 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl"] Jul 24 17:30:27.745103 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:30:27.745076 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5befb0_2d92_49ef_8679_eb19b9cbae19.slice/crio-f3482c97c8d77a31a5029bbde553c976ce61258a9f12d907d0783e6b88345066 WatchSource:0}: Error finding container f3482c97c8d77a31a5029bbde553c976ce61258a9f12d907d0783e6b88345066: Status 404 returned error can't find the container with id f3482c97c8d77a31a5029bbde553c976ce61258a9f12d907d0783e6b88345066 Jul 24 17:30:27.917689 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:27.917656 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:28.013570 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:28.013540 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:30:28.321847 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:28.321807 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" event={"ID":"de5befb0-2d92-49ef-8679-eb19b9cbae19","Type":"ContainerStarted","Data":"e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af"} Jul 24 17:30:28.322292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:28.321877 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" event={"ID":"de5befb0-2d92-49ef-8679-eb19b9cbae19","Type":"ContainerStarted","Data":"f3482c97c8d77a31a5029bbde553c976ce61258a9f12d907d0783e6b88345066"} Jul 24 17:30:28.322292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:28.322008 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:30:28.337881 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:28.337831 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" podStartSLOduration=0.89050308 podStartE2EDuration="1.337812087s" totalImagesPullingTime="447.309007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:30:28.337263125 +0000 UTC m=+1645.835665675" watchObservedRunningTime="2026-07-24 17:30:28.337812087 +0000 UTC m=+1645.836214633" Jul 24 17:30:34.333465 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:30:34.333398 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:31:02.629462 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.629422 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl"] Jul 24 17:31:02.630015 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.629674 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" podUID="de5befb0-2d92-49ef-8679-eb19b9cbae19" containerName="maas-api" containerID="cri-o://e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af" gracePeriod=30 Jul 24 17:31:02.667287 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.667247 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:02.910171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.910143 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:31:02.957547 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.957517 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5befb0-2d92-49ef-8679-eb19b9cbae19-maas-api-tls\") pod \"de5befb0-2d92-49ef-8679-eb19b9cbae19\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " Jul 24 17:31:02.957714 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.957559 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5befb0-2d92-49ef-8679-eb19b9cbae19-openshift-service-ca\") pod \"de5befb0-2d92-49ef-8679-eb19b9cbae19\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " Jul 24 17:31:02.957714 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.957583 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj2b4\" (UniqueName: \"kubernetes.io/projected/de5befb0-2d92-49ef-8679-eb19b9cbae19-kube-api-access-fj2b4\") pod \"de5befb0-2d92-49ef-8679-eb19b9cbae19\" (UID: \"de5befb0-2d92-49ef-8679-eb19b9cbae19\") " Jul 24 17:31:02.958028 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.957978 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5befb0-2d92-49ef-8679-eb19b9cbae19-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de5befb0-2d92-49ef-8679-eb19b9cbae19" (UID: "de5befb0-2d92-49ef-8679-eb19b9cbae19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:31:02.959816 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.959788 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5befb0-2d92-49ef-8679-eb19b9cbae19-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de5befb0-2d92-49ef-8679-eb19b9cbae19" (UID: "de5befb0-2d92-49ef-8679-eb19b9cbae19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:31:02.960226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:02.960191 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5befb0-2d92-49ef-8679-eb19b9cbae19-kube-api-access-fj2b4" (OuterVolumeSpecName: "kube-api-access-fj2b4") pod "de5befb0-2d92-49ef-8679-eb19b9cbae19" (UID: "de5befb0-2d92-49ef-8679-eb19b9cbae19"). InnerVolumeSpecName "kube-api-access-fj2b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:31:03.059179 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.059147 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5befb0-2d92-49ef-8679-eb19b9cbae19-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:31:03.059179 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.059175 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5befb0-2d92-49ef-8679-eb19b9cbae19-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:31:03.059384 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.059190 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj2b4\" (UniqueName: \"kubernetes.io/projected/de5befb0-2d92-49ef-8679-eb19b9cbae19-kube-api-access-fj2b4\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:31:03.504963 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.504931 2552 generic.go:356] "Generic (PLEG): container finished" podID="de5befb0-2d92-49ef-8679-eb19b9cbae19" containerID="e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af" exitCode=0 Jul 24 17:31:03.504963 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.504969 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" event={"ID":"de5befb0-2d92-49ef-8679-eb19b9cbae19","Type":"ContainerDied","Data":"e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af"} Jul 24 17:31:03.505264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.505011 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" event={"ID":"de5befb0-2d92-49ef-8679-eb19b9cbae19","Type":"ContainerDied","Data":"f3482c97c8d77a31a5029bbde553c976ce61258a9f12d907d0783e6b88345066"} Jul 24 17:31:03.505264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.505034 2552 scope.go:117] "RemoveContainer" containerID="e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af" Jul 24 17:31:03.505264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.504970 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl" Jul 24 17:31:03.513467 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.513449 2552 scope.go:117] "RemoveContainer" containerID="e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af" Jul 24 17:31:03.513704 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:31:03.513685 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af\": container with ID starting with e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af not found: ID does not exist" containerID="e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af" Jul 24 17:31:03.513760 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.513712 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af"} err="failed to get container status \"e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af\": rpc error: code = NotFound desc = could not find container \"e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af\": container with ID starting with e9514f446709ab4eb9ea39a221297b3aea0bf6516c2efe3519079c6ffecdd8af not found: ID does not exist" Jul 24 17:31:03.523604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.523577 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl"] Jul 24 17:31:03.525683 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.525660 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-1f7abc01-7f78558bc5-gn4pl"] Jul 24 17:31:03.845726 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:03.845640 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:04.965571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:04.965537 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de5befb0-2d92-49ef-8679-eb19b9cbae19" path="/var/lib/kubelet/pods/de5befb0-2d92-49ef-8679-eb19b9cbae19/volumes" Jul 24 17:31:07.269821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:07.269789 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:07.428002 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:07.427964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:07.664642 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:07.664607 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:07.938848 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:07.938771 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:08.270703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:08.270595 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:08.924429 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:08.924366 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:10.341292 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.341256 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:10.424131 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.424094 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:10.433131 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.433096 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr"] Jul 24 17:31:10.433800 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.433780 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" Jul 24 17:31:10.433862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.433806 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f1ae3e-4725-487e-86da-5e0b38194e6b" containerName="istio-proxy" Jul 24 17:31:10.433862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.433823 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de5befb0-2d92-49ef-8679-eb19b9cbae19" containerName="maas-api" Jul 24 17:31:10.433862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.433833 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="de5befb0-2d92-49ef-8679-eb19b9cbae19" containerName="maas-api" Jul 24 17:31:10.433972 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.433944 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="de5befb0-2d92-49ef-8679-eb19b9cbae19" containerName="maas-api" Jul 24 17:31:10.437582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.437560 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.440069 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.440048 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-ait-1f216bac\"" Jul 24 17:31:10.446641 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.446617 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr"] Jul 24 17:31:10.494630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.494595 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:10.538391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.538353 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329503d9-101e-4950-88cb-0ca65e72bd24-openshift-service-ca\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.538583 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.538534 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.538639 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.538588 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/329503d9-101e-4950-88cb-0ca65e72bd24-kube-api-access-ggxcl\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.639458 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.639365 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.639458 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.639435 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/329503d9-101e-4950-88cb-0ca65e72bd24-kube-api-access-ggxcl\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.639663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.639482 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329503d9-101e-4950-88cb-0ca65e72bd24-openshift-service-ca\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.639663 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:31:10.639545 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-ait-1f216bac: secret "maas-api-serving-cert-e2e-ait-1f216bac" not found Jul 24 17:31:10.639663 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:31:10.639633 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls podName:329503d9-101e-4950-88cb-0ca65e72bd24 nodeName:}" failed. No retries permitted until 2026-07-24 17:31:11.13961272 +0000 UTC m=+1688.638015250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls") pod "maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" (UID: "329503d9-101e-4950-88cb-0ca65e72bd24") : secret "maas-api-serving-cert-e2e-ait-1f216bac" not found Jul 24 17:31:10.640115 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.640098 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329503d9-101e-4950-88cb-0ca65e72bd24-openshift-service-ca\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.655861 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.655830 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/329503d9-101e-4950-88cb-0ca65e72bd24-kube-api-access-ggxcl\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:10.755069 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:10.755037 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:11.024075 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:11.024034 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:11.116402 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:11.116366 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:11.146166 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:11.146133 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:11.148617 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:11.148588 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls\") pod \"maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:11.352844 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:11.352766 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:11.493962 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:11.493888 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr"] Jul 24 17:31:11.496490 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:31:11.496462 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329503d9_101e_4950_88cb_0ca65e72bd24.slice/crio-d9bb1ea0f2a33e5659168b4174c8654c8860259dbd574cb5ca91a58ee911aba9 WatchSource:0}: Error finding container d9bb1ea0f2a33e5659168b4174c8654c8860259dbd574cb5ca91a58ee911aba9: Status 404 returned error can't find the container with id d9bb1ea0f2a33e5659168b4174c8654c8860259dbd574cb5ca91a58ee911aba9 Jul 24 17:31:11.551277 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:11.551245 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" event={"ID":"329503d9-101e-4950-88cb-0ca65e72bd24","Type":"ContainerStarted","Data":"d9bb1ea0f2a33e5659168b4174c8654c8860259dbd574cb5ca91a58ee911aba9"} Jul 24 17:31:12.558046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:12.558001 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" event={"ID":"329503d9-101e-4950-88cb-0ca65e72bd24","Type":"ContainerStarted","Data":"691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f"} Jul 24 17:31:12.558454 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:12.558190 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:12.577984 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:12.577937 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" podStartSLOduration=2.195291473 podStartE2EDuration="2.577921473s" totalImagesPullingTime="382.63ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:31:12.575928545 +0000 UTC m=+1690.074331116" watchObservedRunningTime="2026-07-24 17:31:12.577921473 +0000 UTC m=+1690.076324019" Jul 24 17:31:18.568625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:18.568594 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:46.825311 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:46.825276 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr"] Jul 24 17:31:46.826148 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:46.826111 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" podUID="329503d9-101e-4950-88cb-0ca65e72bd24" containerName="maas-api" containerID="cri-o://691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f" gracePeriod=30 Jul 24 17:31:46.859665 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:46.859639 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:46.990058 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:46.990019 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:47.089606 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.089546 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:47.110254 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.110227 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:47.152614 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.152586 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls\") pod \"329503d9-101e-4950-88cb-0ca65e72bd24\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " Jul 24 17:31:47.152809 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.152634 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/329503d9-101e-4950-88cb-0ca65e72bd24-kube-api-access-ggxcl\") pod \"329503d9-101e-4950-88cb-0ca65e72bd24\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " Jul 24 17:31:47.152809 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.152714 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329503d9-101e-4950-88cb-0ca65e72bd24-openshift-service-ca\") pod \"329503d9-101e-4950-88cb-0ca65e72bd24\" (UID: \"329503d9-101e-4950-88cb-0ca65e72bd24\") " Jul 24 17:31:47.153163 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.153126 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329503d9-101e-4950-88cb-0ca65e72bd24-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "329503d9-101e-4950-88cb-0ca65e72bd24" (UID: "329503d9-101e-4950-88cb-0ca65e72bd24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:31:47.154928 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.154898 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329503d9-101e-4950-88cb-0ca65e72bd24-kube-api-access-ggxcl" (OuterVolumeSpecName: "kube-api-access-ggxcl") pod "329503d9-101e-4950-88cb-0ca65e72bd24" (UID: "329503d9-101e-4950-88cb-0ca65e72bd24"). InnerVolumeSpecName "kube-api-access-ggxcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:31:47.155025 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.154941 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "329503d9-101e-4950-88cb-0ca65e72bd24" (UID: "329503d9-101e-4950-88cb-0ca65e72bd24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:31:47.253660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.253621 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/329503d9-101e-4950-88cb-0ca65e72bd24-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:31:47.253660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.253653 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/329503d9-101e-4950-88cb-0ca65e72bd24-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:31:47.253660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.253663 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxcl\" (UniqueName: \"kubernetes.io/projected/329503d9-101e-4950-88cb-0ca65e72bd24-kube-api-access-ggxcl\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:31:47.752805 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.752767 2552 generic.go:356] "Generic (PLEG): container finished" podID="329503d9-101e-4950-88cb-0ca65e72bd24" containerID="691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f" exitCode=0 Jul 24 17:31:47.752984 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.752814 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" Jul 24 17:31:47.752984 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.752847 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" event={"ID":"329503d9-101e-4950-88cb-0ca65e72bd24","Type":"ContainerDied","Data":"691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f"} Jul 24 17:31:47.752984 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.752908 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr" event={"ID":"329503d9-101e-4950-88cb-0ca65e72bd24","Type":"ContainerDied","Data":"d9bb1ea0f2a33e5659168b4174c8654c8860259dbd574cb5ca91a58ee911aba9"} Jul 24 17:31:47.752984 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.752930 2552 scope.go:117] "RemoveContainer" containerID="691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f" Jul 24 17:31:47.762565 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.762542 2552 scope.go:117] "RemoveContainer" containerID="691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f" Jul 24 17:31:47.762851 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:31:47.762825 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f\": container with ID starting with 691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f not found: ID does not exist" containerID="691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f" Jul 24 17:31:47.762917 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.762864 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f"} err="failed to get container status \"691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f\": rpc error: code = NotFound desc = could not find container \"691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f\": container with ID starting with 691bca6bda8feb45d318878a33ec0237f7bdea3c41d1cdcefa5097e41ff9e53f not found: ID does not exist" Jul 24 17:31:47.776313 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.776286 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr"] Jul 24 17:31:47.780660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:47.780637 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f216bac-75ccb74875-zh5pr"] Jul 24 17:31:48.964263 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:48.964220 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="329503d9-101e-4950-88cb-0ca65e72bd24" path="/var/lib/kubelet/pods/329503d9-101e-4950-88cb-0ca65e72bd24/volumes" Jul 24 17:31:51.346871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:51.346835 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:52.568535 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:52.568501 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:52.803780 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:52.803744 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:53.028490 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:53.028460 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:53.346046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:53.345957 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:53.568830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:53.568795 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:55.637918 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.637888 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:55.737590 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.737557 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:55.823087 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.823052 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj"] Jul 24 17:31:55.823923 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.823902 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="329503d9-101e-4950-88cb-0ca65e72bd24" containerName="maas-api" Jul 24 17:31:55.824054 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.824043 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="329503d9-101e-4950-88cb-0ca65e72bd24" containerName="maas-api" Jul 24 17:31:55.824287 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.824266 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="329503d9-101e-4950-88cb-0ca65e72bd24" containerName="maas-api" Jul 24 17:31:55.828983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.828962 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:55.831168 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.831144 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-derive-deb90cc5\"" Jul 24 17:31:55.835897 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.835876 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj"] Jul 24 17:31:55.862893 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.862862 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:55.932364 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.932329 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twv9\" (UniqueName: \"kubernetes.io/projected/62fa8232-a46e-420a-802e-cfd9de3ba5f1-kube-api-access-9twv9\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:55.932364 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.932367 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fa8232-a46e-420a-802e-cfd9de3ba5f1-maas-api-tls\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:55.932627 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.932386 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fa8232-a46e-420a-802e-cfd9de3ba5f1-openshift-service-ca\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:55.978042 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:55.978010 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:56.033158 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.033115 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twv9\" (UniqueName: \"kubernetes.io/projected/62fa8232-a46e-420a-802e-cfd9de3ba5f1-kube-api-access-9twv9\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:56.033345 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.033171 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fa8232-a46e-420a-802e-cfd9de3ba5f1-maas-api-tls\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:56.033345 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.033191 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fa8232-a46e-420a-802e-cfd9de3ba5f1-openshift-service-ca\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:56.033928 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.033905 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fa8232-a46e-420a-802e-cfd9de3ba5f1-openshift-service-ca\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:56.035668 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.035641 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fa8232-a46e-420a-802e-cfd9de3ba5f1-maas-api-tls\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:56.043075 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.043050 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twv9\" (UniqueName: \"kubernetes.io/projected/62fa8232-a46e-420a-802e-cfd9de3ba5f1-kube-api-access-9twv9\") pod \"maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:56.141291 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.141253 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:56.356104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.356035 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:56.451048 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.450961 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:31:56.499987 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.499961 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj"] Jul 24 17:31:56.501637 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:31:56.501613 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fa8232_a46e_420a_802e_cfd9de3ba5f1.slice/crio-67f9adecd3fe7a2346dfabeddd28eddb3620dbfd0cb0bce148abea512216dc3d WatchSource:0}: Error finding container 67f9adecd3fe7a2346dfabeddd28eddb3620dbfd0cb0bce148abea512216dc3d: Status 404 returned error can't find the container with id 67f9adecd3fe7a2346dfabeddd28eddb3620dbfd0cb0bce148abea512216dc3d Jul 24 17:31:56.801932 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:56.801842 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" event={"ID":"62fa8232-a46e-420a-802e-cfd9de3ba5f1","Type":"ContainerStarted","Data":"67f9adecd3fe7a2346dfabeddd28eddb3620dbfd0cb0bce148abea512216dc3d"} Jul 24 17:31:57.808189 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:57.808145 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" event={"ID":"62fa8232-a46e-420a-802e-cfd9de3ba5f1","Type":"ContainerStarted","Data":"695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d"} Jul 24 17:31:57.808667 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:57.808289 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:31:57.826232 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:31:57.826179 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" podStartSLOduration=2.493445904 podStartE2EDuration="2.826163664s" totalImagesPullingTime="332.71776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:31:57.824579114 +0000 UTC m=+1735.322981685" watchObservedRunningTime="2026-07-24 17:31:57.826163664 +0000 UTC m=+1735.324566211" Jul 24 17:32:03.818247 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:03.818217 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:32:11.250842 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:11.250806 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:30.893690 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:30.893654 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj"] Jul 24 17:32:30.894239 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:30.893902 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" podUID="62fa8232-a46e-420a-802e-cfd9de3ba5f1" containerName="maas-api" containerID="cri-o://695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d" gracePeriod=30 Jul 24 17:32:30.921160 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:30.921122 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:30.993600 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:30.993567 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:31.063459 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.063401 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:31.149796 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.149736 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:32:31.252067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.252033 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fa8232-a46e-420a-802e-cfd9de3ba5f1-openshift-service-ca\") pod \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " Jul 24 17:32:31.252238 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.252076 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9twv9\" (UniqueName: \"kubernetes.io/projected/62fa8232-a46e-420a-802e-cfd9de3ba5f1-kube-api-access-9twv9\") pod \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " Jul 24 17:32:31.252238 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.252103 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fa8232-a46e-420a-802e-cfd9de3ba5f1-maas-api-tls\") pod \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\" (UID: \"62fa8232-a46e-420a-802e-cfd9de3ba5f1\") " Jul 24 17:32:31.252477 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.252448 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fa8232-a46e-420a-802e-cfd9de3ba5f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62fa8232-a46e-420a-802e-cfd9de3ba5f1" (UID: "62fa8232-a46e-420a-802e-cfd9de3ba5f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:32:31.254193 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.254171 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fa8232-a46e-420a-802e-cfd9de3ba5f1-kube-api-access-9twv9" (OuterVolumeSpecName: "kube-api-access-9twv9") pod "62fa8232-a46e-420a-802e-cfd9de3ba5f1" (UID: "62fa8232-a46e-420a-802e-cfd9de3ba5f1"). InnerVolumeSpecName "kube-api-access-9twv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:32:31.254296 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.254203 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62fa8232-a46e-420a-802e-cfd9de3ba5f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62fa8232-a46e-420a-802e-cfd9de3ba5f1" (UID: "62fa8232-a46e-420a-802e-cfd9de3ba5f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:32:31.352922 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.352891 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62fa8232-a46e-420a-802e-cfd9de3ba5f1-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:32:31.352922 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.352919 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9twv9\" (UniqueName: \"kubernetes.io/projected/62fa8232-a46e-420a-802e-cfd9de3ba5f1-kube-api-access-9twv9\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:32:31.352922 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.352929 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62fa8232-a46e-420a-802e-cfd9de3ba5f1-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:32:31.995023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.994989 2552 generic.go:356] "Generic (PLEG): container finished" podID="62fa8232-a46e-420a-802e-cfd9de3ba5f1" containerID="695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d" exitCode=0 Jul 24 17:32:31.995542 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.995023 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" event={"ID":"62fa8232-a46e-420a-802e-cfd9de3ba5f1","Type":"ContainerDied","Data":"695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d"} Jul 24 17:32:31.995542 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.995077 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" event={"ID":"62fa8232-a46e-420a-802e-cfd9de3ba5f1","Type":"ContainerDied","Data":"67f9adecd3fe7a2346dfabeddd28eddb3620dbfd0cb0bce148abea512216dc3d"} Jul 24 17:32:31.995542 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.995097 2552 scope.go:117] "RemoveContainer" containerID="695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d" Jul 24 17:32:31.995542 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:31.995034 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj" Jul 24 17:32:32.004140 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:32.004123 2552 scope.go:117] "RemoveContainer" containerID="695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d" Jul 24 17:32:32.004404 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:32:32.004387 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d\": container with ID starting with 695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d not found: ID does not exist" containerID="695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d" Jul 24 17:32:32.004487 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:32.004430 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d"} err="failed to get container status \"695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d\": rpc error: code = NotFound desc = could not find container \"695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d\": container with ID starting with 695ed62395211200c8a9f7fcd65825a2561d4503e6925852df70dd5221e6149d not found: ID does not exist" Jul 24 17:32:32.018321 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:32.018298 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj"] Jul 24 17:32:32.022842 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:32.022820 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-deb90cc5-5f9d9dfb89-gc5sj"] Jul 24 17:32:32.964977 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:32.964945 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62fa8232-a46e-420a-802e-cfd9de3ba5f1" path="/var/lib/kubelet/pods/62fa8232-a46e-420a-802e-cfd9de3ba5f1/volumes" Jul 24 17:32:35.548306 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:35.548265 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:35.703660 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:35.703620 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:35.950625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:35.950591 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:36.522500 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:36.522468 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:36.626226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:36.626193 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:38.105100 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:38.105064 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:44.124141 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:44.124101 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:44.228897 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:44.228866 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:44.325197 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:44.325160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:44.424986 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:44.424952 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:44.858110 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:44.858024 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:32:48.302578 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:32:48.302546 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:03.044273 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:03.044242 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:33:03.052288 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:03.052265 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:33:10.814884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:10.814839 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:17.433632 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:17.433595 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:17.650183 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:17.650144 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:27.607629 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:27.607591 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:27.708526 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:27.708494 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:33.522646 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:33.522614 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:39.490509 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:39.490474 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:39.637745 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:39.637708 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:39.737672 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:39.737641 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:50.231720 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:50.231674 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:51.166604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:51.166563 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:57.281842 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:57.281809 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:57.396777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:57.396737 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:57.493189 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:57.493153 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:57.623519 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:57.623432 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:58.454655 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:58.454603 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:58.549482 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:58.549399 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:33:58.655605 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:33:58.655551 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:05.399244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:05.399210 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:12.054995 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:12.054957 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:12.153511 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:12.153477 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:12.725866 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:12.725831 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:19.075555 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:19.075522 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:19.170486 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:19.170444 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:19.273477 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:19.273441 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:19.602948 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:19.602920 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:19.706918 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:19.706878 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:22.935636 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:22.935591 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:23.023125 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.023091 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:23.128163 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.128128 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj"] Jul 24 17:34:23.129030 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.129001 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62fa8232-a46e-420a-802e-cfd9de3ba5f1" containerName="maas-api" Jul 24 17:34:23.129030 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.129027 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="62fa8232-a46e-420a-802e-cfd9de3ba5f1" containerName="maas-api" Jul 24 17:34:23.129228 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.129157 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="62fa8232-a46e-420a-802e-cfd9de3ba5f1" containerName="maas-api" Jul 24 17:34:23.131846 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.131828 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.134219 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.134200 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-mt-0bdb575b\"" Jul 24 17:34:23.142211 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.142187 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj"] Jul 24 17:34:23.162701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.162658 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:23.317932 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.317832 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.318125 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.317947 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7qdm\" (UniqueName: \"kubernetes.io/projected/e6096d96-1508-4592-85bb-fdf90910bf35-kube-api-access-f7qdm\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.318125 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.318032 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6096d96-1508-4592-85bb-fdf90910bf35-openshift-service-ca\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.418819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.418774 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6096d96-1508-4592-85bb-fdf90910bf35-openshift-service-ca\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.419012 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.418859 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.419012 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.418916 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7qdm\" (UniqueName: \"kubernetes.io/projected/e6096d96-1508-4592-85bb-fdf90910bf35-kube-api-access-f7qdm\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.419928 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:34:23.419901 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-mt-0bdb575b: secret "maas-api-serving-cert-e2e-mt-0bdb575b" not found Jul 24 17:34:23.419928 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.419918 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6096d96-1508-4592-85bb-fdf90910bf35-openshift-service-ca\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.420120 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:34:23.419989 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls podName:e6096d96-1508-4592-85bb-fdf90910bf35 nodeName:}" failed. No retries permitted until 2026-07-24 17:34:23.919974813 +0000 UTC m=+1881.418377340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls") pod "maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" (UID: "e6096d96-1508-4592-85bb-fdf90910bf35") : secret "maas-api-serving-cert-e2e-mt-0bdb575b" not found Jul 24 17:34:23.430701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.430666 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7qdm\" (UniqueName: \"kubernetes.io/projected/e6096d96-1508-4592-85bb-fdf90910bf35-kube-api-access-f7qdm\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.683300 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.683264 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:23.822898 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.822858 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:23.914792 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.914758 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:23.923094 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.923063 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:23.925852 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:23.925827 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls\") pod \"maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:24.044884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:24.044789 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:24.391604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:24.391529 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj"] Jul 24 17:34:24.395209 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:34:24.395179 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6096d96_1508_4592_85bb_fdf90910bf35.slice/crio-91de51bc700b9741cd6f8ce43ae1d2ea052c05854bed3375ef56449872b880be WatchSource:0}: Error finding container 91de51bc700b9741cd6f8ce43ae1d2ea052c05854bed3375ef56449872b880be: Status 404 returned error can't find the container with id 91de51bc700b9741cd6f8ce43ae1d2ea052c05854bed3375ef56449872b880be Jul 24 17:34:24.581158 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:24.581118 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" event={"ID":"e6096d96-1508-4592-85bb-fdf90910bf35","Type":"ContainerStarted","Data":"91de51bc700b9741cd6f8ce43ae1d2ea052c05854bed3375ef56449872b880be"} Jul 24 17:34:25.587815 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:25.587733 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" event={"ID":"e6096d96-1508-4592-85bb-fdf90910bf35","Type":"ContainerStarted","Data":"d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a"} Jul 24 17:34:25.588194 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:25.587815 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:25.604849 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:25.604803 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" podStartSLOduration=2.121148335 podStartE2EDuration="2.604787286s" totalImagesPullingTime="483.638951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:34:25.60391483 +0000 UTC m=+1883.102317378" watchObservedRunningTime="2026-07-24 17:34:25.604787286 +0000 UTC m=+1883.103189831" Jul 24 17:34:27.333397 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:27.333357 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:28.061390 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:28.061360 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:28.197066 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:28.197033 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:28.444671 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:28.444638 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:31.598588 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:31.598552 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:34.219621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:34.219581 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:34.321156 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:34.321123 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:45.259995 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.259955 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj"] Jul 24 17:34:45.260757 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.260209 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" podUID="e6096d96-1508-4592-85bb-fdf90910bf35" containerName="maas-api" containerID="cri-o://d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a" gracePeriod=30 Jul 24 17:34:45.354357 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.354318 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:45.514895 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.514832 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:45.543862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.543832 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:45.611900 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.611870 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls\") pod \"e6096d96-1508-4592-85bb-fdf90910bf35\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " Jul 24 17:34:45.612084 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.611931 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7qdm\" (UniqueName: \"kubernetes.io/projected/e6096d96-1508-4592-85bb-fdf90910bf35-kube-api-access-f7qdm\") pod \"e6096d96-1508-4592-85bb-fdf90910bf35\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " Jul 24 17:34:45.612084 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.611951 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6096d96-1508-4592-85bb-fdf90910bf35-openshift-service-ca\") pod \"e6096d96-1508-4592-85bb-fdf90910bf35\" (UID: \"e6096d96-1508-4592-85bb-fdf90910bf35\") " Jul 24 17:34:45.612491 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.612442 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6096d96-1508-4592-85bb-fdf90910bf35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e6096d96-1508-4592-85bb-fdf90910bf35" (UID: "e6096d96-1508-4592-85bb-fdf90910bf35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:34:45.614365 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.614336 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e6096d96-1508-4592-85bb-fdf90910bf35" (UID: "e6096d96-1508-4592-85bb-fdf90910bf35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:34:45.614499 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.614371 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6096d96-1508-4592-85bb-fdf90910bf35-kube-api-access-f7qdm" (OuterVolumeSpecName: "kube-api-access-f7qdm") pod "e6096d96-1508-4592-85bb-fdf90910bf35" (UID: "e6096d96-1508-4592-85bb-fdf90910bf35"). InnerVolumeSpecName "kube-api-access-f7qdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:34:45.684883 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.684851 2552 generic.go:356] "Generic (PLEG): container finished" podID="e6096d96-1508-4592-85bb-fdf90910bf35" containerID="d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a" exitCode=0 Jul 24 17:34:45.685062 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.684899 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" Jul 24 17:34:45.685062 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.684934 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" event={"ID":"e6096d96-1508-4592-85bb-fdf90910bf35","Type":"ContainerDied","Data":"d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a"} Jul 24 17:34:45.685062 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.685010 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj" event={"ID":"e6096d96-1508-4592-85bb-fdf90910bf35","Type":"ContainerDied","Data":"91de51bc700b9741cd6f8ce43ae1d2ea052c05854bed3375ef56449872b880be"} Jul 24 17:34:45.685062 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.685036 2552 scope.go:117] "RemoveContainer" containerID="d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a" Jul 24 17:34:45.694934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.694903 2552 scope.go:117] "RemoveContainer" containerID="d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a" Jul 24 17:34:45.695244 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:34:45.695219 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a\": container with ID starting with d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a not found: ID does not exist" containerID="d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a" Jul 24 17:34:45.695338 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.695259 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a"} err="failed to get container status \"d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a\": rpc error: code = NotFound desc = could not find container \"d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a\": container with ID starting with d320aacc3de0c262e46266c7da6a6135277c6558f8e8e4098c54a03a8346885a not found: ID does not exist" Jul 24 17:34:45.711592 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.711564 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj"] Jul 24 17:34:45.713271 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.713249 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6096d96-1508-4592-85bb-fdf90910bf35-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:34:45.713271 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.713272 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7qdm\" (UniqueName: \"kubernetes.io/projected/e6096d96-1508-4592-85bb-fdf90910bf35-kube-api-access-f7qdm\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:34:45.713390 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.713282 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6096d96-1508-4592-85bb-fdf90910bf35-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:34:45.713804 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:45.713785 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0bdb575b-d97f8db95-2zjlj"] Jul 24 17:34:46.965626 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:46.965596 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6096d96-1508-4592-85bb-fdf90910bf35" path="/var/lib/kubelet/pods/e6096d96-1508-4592-85bb-fdf90910bf35/volumes" Jul 24 17:34:49.676870 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:49.676841 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:50.093940 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:50.093864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:56.075987 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:56.075927 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:56.177607 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:56.177574 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:56.349636 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:56.349546 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:56.576351 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:56.576291 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:56.725049 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:56.725015 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:56.815125 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:56.815095 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:34:56.932899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:34:56.932867 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:03.214046 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:03.214007 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:03.307865 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:03.307827 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:03.548480 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:03.548365 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:03.644869 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:03.644803 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:09.528893 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:09.528837 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:09.595374 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:09.595343 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:16.329032 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:16.328996 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:16.643684 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:16.643587 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:16.810468 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:16.810437 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:20.082250 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.082213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:20.161530 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.161496 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g"] Jul 24 17:35:20.162100 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.162085 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6096d96-1508-4592-85bb-fdf90910bf35" containerName="maas-api" Jul 24 17:35:20.162171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.162103 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6096d96-1508-4592-85bb-fdf90910bf35" containerName="maas-api" Jul 24 17:35:20.162234 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.162205 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6096d96-1508-4592-85bb-fdf90910bf35" containerName="maas-api" Jul 24 17:35:20.165807 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.165786 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.168497 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.168478 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-mt-4fe99dd6\"" Jul 24 17:35:20.176497 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.176461 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:20.177390 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.177367 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g"] Jul 24 17:35:20.247237 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.247205 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:20.324759 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.324718 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e171a18-acf5-4f75-b40a-abb4fd47032f-maas-api-tls\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.324946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.324804 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e171a18-acf5-4f75-b40a-abb4fd47032f-openshift-service-ca\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.324946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.324874 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgk6\" (UniqueName: \"kubernetes.io/projected/7e171a18-acf5-4f75-b40a-abb4fd47032f-kube-api-access-lfgk6\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.401477 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.401374 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:20.425420 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.425387 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e171a18-acf5-4f75-b40a-abb4fd47032f-openshift-service-ca\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.425571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.425479 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgk6\" (UniqueName: \"kubernetes.io/projected/7e171a18-acf5-4f75-b40a-abb4fd47032f-kube-api-access-lfgk6\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.425571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.425522 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e171a18-acf5-4f75-b40a-abb4fd47032f-maas-api-tls\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.426066 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.426042 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e171a18-acf5-4f75-b40a-abb4fd47032f-openshift-service-ca\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.427948 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.427925 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e171a18-acf5-4f75-b40a-abb4fd47032f-maas-api-tls\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.434197 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.434174 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgk6\" (UniqueName: \"kubernetes.io/projected/7e171a18-acf5-4f75-b40a-abb4fd47032f-kube-api-access-lfgk6\") pod \"maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.476763 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.476736 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:20.618536 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.618504 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g"] Jul 24 17:35:20.619909 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:35:20.619883 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e171a18_acf5_4f75_b40a_abb4fd47032f.slice/crio-db99c1c5dbbdf32ff827595c5328050247cc0b851ff1243a0f7228e40b046101 WatchSource:0}: Error finding container db99c1c5dbbdf32ff827595c5328050247cc0b851ff1243a0f7228e40b046101: Status 404 returned error can't find the container with id db99c1c5dbbdf32ff827595c5328050247cc0b851ff1243a0f7228e40b046101 Jul 24 17:35:20.621419 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.621380 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:35:20.820643 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.820612 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:20.865264 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.865222 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" event={"ID":"7e171a18-acf5-4f75-b40a-abb4fd47032f","Type":"ContainerStarted","Data":"db99c1c5dbbdf32ff827595c5328050247cc0b851ff1243a0f7228e40b046101"} Jul 24 17:35:20.921177 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:20.921144 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:21.872582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:21.872547 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" event={"ID":"7e171a18-acf5-4f75-b40a-abb4fd47032f","Type":"ContainerStarted","Data":"d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0"} Jul 24 17:35:21.873018 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:21.872680 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:21.891528 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:21.891482 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" podStartSLOduration=1.466858306 podStartE2EDuration="1.891467817s" totalImagesPullingTime="424.609511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:35:21.890081702 +0000 UTC m=+1939.388484247" watchObservedRunningTime="2026-07-24 17:35:21.891467817 +0000 UTC m=+1939.389870363" Jul 24 17:35:24.554339 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:24.554308 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:27.882290 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:27.882256 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:35:29.920982 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:29.920931 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:34.301488 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:34.301457 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:35:48.816860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:35:48.816819 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:36:29.960749 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:29.960710 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g"] Jul 24 17:36:29.961300 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:29.960901 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" podUID="7e171a18-acf5-4f75-b40a-abb4fd47032f" containerName="maas-api" containerID="cri-o://d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0" gracePeriod=30 Jul 24 17:36:29.992690 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:29.992663 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:36:30.225802 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.225726 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:36:30.228929 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.228898 2552 generic.go:356] "Generic (PLEG): container finished" podID="7e171a18-acf5-4f75-b40a-abb4fd47032f" containerID="d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0" exitCode=0 Jul 24 17:36:30.229083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.228938 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" event={"ID":"7e171a18-acf5-4f75-b40a-abb4fd47032f","Type":"ContainerDied","Data":"d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0"} Jul 24 17:36:30.229083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.228945 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" Jul 24 17:36:30.229083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.228989 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g" event={"ID":"7e171a18-acf5-4f75-b40a-abb4fd47032f","Type":"ContainerDied","Data":"db99c1c5dbbdf32ff827595c5328050247cc0b851ff1243a0f7228e40b046101"} Jul 24 17:36:30.229083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.229020 2552 scope.go:117] "RemoveContainer" containerID="d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0" Jul 24 17:36:30.237905 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.237884 2552 scope.go:117] "RemoveContainer" containerID="d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0" Jul 24 17:36:30.238252 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:36:30.238232 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0\": container with ID starting with d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0 not found: ID does not exist" containerID="d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0" Jul 24 17:36:30.238325 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.238262 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0"} err="failed to get container status \"d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0\": rpc error: code = NotFound desc = could not find container \"d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0\": container with ID starting with d083f40dea9afc5cc55146d499030bd5d6259b854a3322383e7f21e9817f1bd0 not found: ID does not exist" Jul 24 17:36:30.383741 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.383710 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e171a18-acf5-4f75-b40a-abb4fd47032f-maas-api-tls\") pod \"7e171a18-acf5-4f75-b40a-abb4fd47032f\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " Jul 24 17:36:30.383953 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.383764 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfgk6\" (UniqueName: \"kubernetes.io/projected/7e171a18-acf5-4f75-b40a-abb4fd47032f-kube-api-access-lfgk6\") pod \"7e171a18-acf5-4f75-b40a-abb4fd47032f\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " Jul 24 17:36:30.383953 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.383892 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e171a18-acf5-4f75-b40a-abb4fd47032f-openshift-service-ca\") pod \"7e171a18-acf5-4f75-b40a-abb4fd47032f\" (UID: \"7e171a18-acf5-4f75-b40a-abb4fd47032f\") " Jul 24 17:36:30.384222 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.384197 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e171a18-acf5-4f75-b40a-abb4fd47032f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e171a18-acf5-4f75-b40a-abb4fd47032f" (UID: "7e171a18-acf5-4f75-b40a-abb4fd47032f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:36:30.385947 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.385919 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e171a18-acf5-4f75-b40a-abb4fd47032f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e171a18-acf5-4f75-b40a-abb4fd47032f" (UID: "7e171a18-acf5-4f75-b40a-abb4fd47032f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:36:30.386072 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.386051 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e171a18-acf5-4f75-b40a-abb4fd47032f-kube-api-access-lfgk6" (OuterVolumeSpecName: "kube-api-access-lfgk6") pod "7e171a18-acf5-4f75-b40a-abb4fd47032f" (UID: "7e171a18-acf5-4f75-b40a-abb4fd47032f"). InnerVolumeSpecName "kube-api-access-lfgk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:36:30.485006 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.484911 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e171a18-acf5-4f75-b40a-abb4fd47032f-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:36:30.485006 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.484944 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfgk6\" (UniqueName: \"kubernetes.io/projected/7e171a18-acf5-4f75-b40a-abb4fd47032f-kube-api-access-lfgk6\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:36:30.485006 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.484959 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e171a18-acf5-4f75-b40a-abb4fd47032f-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:36:30.551142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.551101 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g"] Jul 24 17:36:30.553380 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.553353 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4fe99dd6-66c98cddf9-lrx9g"] Jul 24 17:36:30.965066 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:30.965038 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e171a18-acf5-4f75-b40a-abb4fd47032f" path="/var/lib/kubelet/pods/7e171a18-acf5-4f75-b40a-abb4fd47032f/volumes" Jul 24 17:36:31.279775 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:31.279682 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:36:34.455247 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:34.455204 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:36:37.976654 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:37.976619 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:36:43.014896 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:36:43.014860 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:19.698370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:19.698334 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:20.179428 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:20.179378 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:25.802874 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:25.802783 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:25.902860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:25.902828 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:26.112983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:26.112901 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:26.334459 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:26.334380 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:27.455658 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:27.455617 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:27.602798 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:27.602762 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:27.711203 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:27.711106 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:28.453848 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:28.453816 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:28.537286 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:28.537252 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:28.637778 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:28.637727 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:30.103242 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:30.103201 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:36.048998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:36.048962 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:36.149778 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:36.149744 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:36.778052 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:36.778019 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:44.103496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:44.103426 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:44.284670 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:44.284631 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:44.400184 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:44.399993 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:54.958442 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:54.958370 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:37:55.326157 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:37:55.326067 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:03.074446 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:03.074394 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:38:03.083361 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:03.083338 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:38:16.283749 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:16.283713 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:22.412961 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:22.412894 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:22.514936 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:22.514881 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:22.850466 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:22.850360 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:23.156699 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:23.156668 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:23.538089 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:23.537956 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:26.021867 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.021831 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:26.101981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.101944 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t"] Jul 24 17:38:26.102428 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.102390 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e171a18-acf5-4f75-b40a-abb4fd47032f" containerName="maas-api" Jul 24 17:38:26.102428 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.102422 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e171a18-acf5-4f75-b40a-abb4fd47032f" containerName="maas-api" Jul 24 17:38:26.102604 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.102509 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e171a18-acf5-4f75-b40a-abb4fd47032f" containerName="maas-api" Jul 24 17:38:26.105692 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.105672 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.108248 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.108225 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-inf-a-0d8e36\"" Jul 24 17:38:26.114277 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.114254 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:26.117345 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.117315 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t"] Jul 24 17:38:26.158763 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.158721 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmshh\" (UniqueName: \"kubernetes.io/projected/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-kube-api-access-nmshh\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.158936 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.158785 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.158936 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.158884 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.194314 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.194275 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:26.260294 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.260256 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.260488 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.260300 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmshh\" (UniqueName: \"kubernetes.io/projected/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-kube-api-access-nmshh\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.260488 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.260353 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.260488 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:38:26.260445 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-inf-a-0d8e36: secret "maas-api-serving-cert-e2e-inf-a-0d8e36" not found Jul 24 17:38:26.260628 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:38:26.260524 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls podName:40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d nodeName:}" failed. No retries permitted until 2026-07-24 17:38:26.760501477 +0000 UTC m=+2124.258904005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls") pod "maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" (UID: "40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d") : secret "maas-api-serving-cert-e2e-inf-a-0d8e36" not found Jul 24 17:38:26.261090 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.261071 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.270997 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.270966 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmshh\" (UniqueName: \"kubernetes.io/projected/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-kube-api-access-nmshh\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.422627 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.422589 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:26.698378 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.698276 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:26.764388 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.764357 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.766885 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.766852 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls\") pod \"maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:26.798628 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:26.798596 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:27.020230 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:27.020118 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:27.164265 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:27.164242 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t"] Jul 24 17:38:27.166528 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:38:27.166500 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b06ed8_93d1_4537_bdbb_6dfdbf40ba1d.slice/crio-ee6a64b59b4ff00e3f640004ac0381b629a26fdfba4ddfa1e45c4091ec738f90 WatchSource:0}: Error finding container ee6a64b59b4ff00e3f640004ac0381b629a26fdfba4ddfa1e45c4091ec738f90: Status 404 returned error can't find the container with id ee6a64b59b4ff00e3f640004ac0381b629a26fdfba4ddfa1e45c4091ec738f90 Jul 24 17:38:27.820679 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:27.820629 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" event={"ID":"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d","Type":"ContainerStarted","Data":"9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef"} Jul 24 17:38:27.820846 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:27.820723 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" event={"ID":"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d","Type":"ContainerStarted","Data":"ee6a64b59b4ff00e3f640004ac0381b629a26fdfba4ddfa1e45c4091ec738f90"} Jul 24 17:38:28.825829 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:28.825787 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:28.844535 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:28.844477 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" podStartSLOduration=2.26981923 podStartE2EDuration="2.844459709s" totalImagesPullingTime="574.640479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:28.842752426 +0000 UTC m=+2126.341154972" watchObservedRunningTime="2026-07-24 17:38:28.844459709 +0000 UTC m=+2126.342862252" Jul 24 17:38:30.541506 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:30.541459 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:30.905700 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:30.905615 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:31.212724 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.212692 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7"] Jul 24 17:38:31.216538 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.216515 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.221023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.220992 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-746c88-openshift-default-dockercfg-z5q4c\"" Jul 24 17:38:31.229330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.229303 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7"] Jul 24 17:38:31.248678 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.248614 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:31.309786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.309747 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbgw\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-kube-api-access-9tbgw\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.309965 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.309798 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-certs\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.309965 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.309881 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e606d-646a-4908-af06-c95941cdc2df-istio-podinfo\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.309965 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.309926 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-socket\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.310167 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.309965 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-envoy\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.310167 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.309993 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-istio-token\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.310167 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.310050 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-data\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.310167 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.310083 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e606d-646a-4908-af06-c95941cdc2df-istiod-ca-cert\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.310167 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.310123 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-credential-socket\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.410939 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.410894 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbgw\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-kube-api-access-9tbgw\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.410939 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.410940 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-certs\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411227 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.410967 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e606d-646a-4908-af06-c95941cdc2df-istio-podinfo\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411227 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411147 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-socket\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411227 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411218 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-envoy\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411260 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-istio-token\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411304 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-data\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411329 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e606d-646a-4908-af06-c95941cdc2df-istiod-ca-cert\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411385 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-credential-socket\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411488 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-socket\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411404 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-certs\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411799 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411684 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-data\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.411799 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.411776 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-credential-socket\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.412071 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.412052 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e606d-646a-4908-af06-c95941cdc2df-istiod-ca-cert\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.413497 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.413474 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-envoy\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.413727 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.413707 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e606d-646a-4908-af06-c95941cdc2df-istio-podinfo\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.420553 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.420523 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbgw\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-kube-api-access-9tbgw\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.420963 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.420944 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-istio-token\") pod \"e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.528882 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.528783 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:31.604216 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.604181 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:31.679951 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.679920 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7"] Jul 24 17:38:31.681683 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:38:31.681644 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23e606d_646a_4908_af06_c95941cdc2df.slice/crio-979a791519562a2bc32fc9cbcc683c0c9a0dc4cb601a0708ab962033e49bec83 WatchSource:0}: Error finding container 979a791519562a2bc32fc9cbcc683c0c9a0dc4cb601a0708ab962033e49bec83: Status 404 returned error can't find the container with id 979a791519562a2bc32fc9cbcc683c0c9a0dc4cb601a0708ab962033e49bec83 Jul 24 17:38:31.683979 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.683937 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:38:31.684057 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.684035 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:38:31.684097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.684077 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:38:31.844655 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.844539 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" event={"ID":"f23e606d-646a-4908-af06-c95941cdc2df","Type":"ContainerStarted","Data":"d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56"} Jul 24 17:38:31.844655 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.844610 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" event={"ID":"f23e606d-646a-4908-af06-c95941cdc2df","Type":"ContainerStarted","Data":"979a791519562a2bc32fc9cbcc683c0c9a0dc4cb601a0708ab962033e49bec83"} Jul 24 17:38:31.866944 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:31.866884 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" podStartSLOduration=0.866867192 podStartE2EDuration="866.867192ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:31.866250318 +0000 UTC m=+2129.364652865" watchObservedRunningTime="2026-07-24 17:38:31.866867192 +0000 UTC m=+2129.365269738" Jul 24 17:38:32.529883 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:32.529839 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:32.535027 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:32.534999 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:32.850212 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:32.850125 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:32.851206 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:32.851188 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:33.883562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:33.883525 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:34.009319 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.009279 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp"] Jul 24 17:38:34.012965 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.012943 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.029055 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.029022 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-inf-b-746c88\"" Jul 24 17:38:34.042518 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.042478 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp"] Jul 24 17:38:34.055665 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.055618 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:34.140059 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.139956 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7729d9-e14c-4010-8420-c6080916b1a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.140059 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.140019 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.140272 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.140123 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp68v\" (UniqueName: \"kubernetes.io/projected/db7729d9-e14c-4010-8420-c6080916b1a9-kube-api-access-fp68v\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.149152 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.149123 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:34.240793 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.240758 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7729d9-e14c-4010-8420-c6080916b1a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.240979 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.240810 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.240979 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.240891 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp68v\" (UniqueName: \"kubernetes.io/projected/db7729d9-e14c-4010-8420-c6080916b1a9-kube-api-access-fp68v\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.241064 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:38:34.240983 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-inf-b-746c88: secret "maas-api-serving-cert-e2e-inf-b-746c88" not found Jul 24 17:38:34.241064 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:38:34.241054 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls podName:db7729d9-e14c-4010-8420-c6080916b1a9 nodeName:}" failed. No retries permitted until 2026-07-24 17:38:34.741036558 +0000 UTC m=+2132.239439082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls") pod "maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" (UID: "db7729d9-e14c-4010-8420-c6080916b1a9") : secret "maas-api-serving-cert-e2e-inf-b-746c88" not found Jul 24 17:38:34.241544 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.241527 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7729d9-e14c-4010-8420-c6080916b1a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.250322 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.250299 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp68v\" (UniqueName: \"kubernetes.io/projected/db7729d9-e14c-4010-8420-c6080916b1a9-kube-api-access-fp68v\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.461174 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.461136 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:34.683662 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.683616 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:34.746168 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.746064 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.748568 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.748540 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-746c88-ff9c699-4d6mp\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:34.785152 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.785117 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:34.837684 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.837647 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:38:34.926447 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:34.926385 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:35.105472 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:35.105440 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp"] Jul 24 17:38:35.106158 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:38:35.106132 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb7729d9_e14c_4010_8420_c6080916b1a9.slice/crio-c03c4ba383285dd3f06462165f2d7dfca1f22aeee05cd513ee557b9002ff56b1 WatchSource:0}: Error finding container c03c4ba383285dd3f06462165f2d7dfca1f22aeee05cd513ee557b9002ff56b1: Status 404 returned error can't find the container with id c03c4ba383285dd3f06462165f2d7dfca1f22aeee05cd513ee557b9002ff56b1 Jul 24 17:38:35.867083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:35.867002 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" event={"ID":"db7729d9-e14c-4010-8420-c6080916b1a9","Type":"ContainerStarted","Data":"64ea9e97b0dceb8004157890e43a49799d3d45e95d181de21f29b6ba4356a6bc"} Jul 24 17:38:35.867083 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:35.867056 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" event={"ID":"db7729d9-e14c-4010-8420-c6080916b1a9","Type":"ContainerStarted","Data":"c03c4ba383285dd3f06462165f2d7dfca1f22aeee05cd513ee557b9002ff56b1"} Jul 24 17:38:35.867286 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:35.867127 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:35.888353 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:35.888298 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" podStartSLOduration=2.400402913 podStartE2EDuration="2.888281163s" totalImagesPullingTime="487.87825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:38:35.886167858 +0000 UTC m=+2133.384570404" watchObservedRunningTime="2026-07-24 17:38:35.888281163 +0000 UTC m=+2133.386683708" Jul 24 17:38:38.343865 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:38.343829 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:41.879218 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:41.879189 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:38:47.599898 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:47.599855 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:38:48.706279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.705535 2552 patch_prober.go:28] interesting pod/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:51590->10.132.0.58:15021: read: connection reset by peer" start-of-body= Jul 24 17:38:48.706279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.705616 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": read tcp 10.132.0.2:51590->10.132.0.58:15021: read: connection reset by peer" Jul 24 17:38:48.942514 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.942478 2552 generic.go:356] "Generic (PLEG): container finished" podID="f23e606d-646a-4908-af06-c95941cdc2df" containerID="d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56" exitCode=0 Jul 24 17:38:48.942722 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.942555 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" event={"ID":"f23e606d-646a-4908-af06-c95941cdc2df","Type":"ContainerDied","Data":"d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56"} Jul 24 17:38:48.943008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.942992 2552 scope.go:117] "RemoveContainer" containerID="d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56" Jul 24 17:38:48.944038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.944008 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:38:48.944145 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.944076 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:38:48.944145 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:48.944112 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:38:49.949356 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:49.949321 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" event={"ID":"f23e606d-646a-4908-af06-c95941cdc2df","Type":"ContainerStarted","Data":"a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286"} Jul 24 17:38:50.529016 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:50.528971 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:50.529426 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:50.529376 2552 patch_prober.go:28] interesting pod/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:38:50.529557 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:50.529472 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 17:38:51.529336 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:51.529296 2552 patch_prober.go:28] interesting pod/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:38:51.529951 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:51.529919 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 17:38:52.529738 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:52.529698 2552 patch_prober.go:28] interesting pod/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:38:52.530350 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:52.530318 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 17:38:53.529945 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:53.529902 2552 patch_prober.go:28] interesting pod/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 24 17:38:53.530476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:53.529967 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 24 17:38:54.533995 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:54.533963 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:54.534564 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:54.534435 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:38:54.535337 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:38:54.535310 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:39:07.003638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:39:07.003551 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:39:58.129664 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:39:58.129626 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:39:59.142115 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:39:59.142083 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:09.748432 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:09.748364 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t"] Jul 24 17:40:09.749355 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:09.749293 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" podUID="40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" containerName="maas-api" containerID="cri-o://9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef" gracePeriod=30 Jul 24 17:40:09.818976 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:09.818939 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:09.831614 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:09.831574 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" podUID="40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.57:8443/health\": dial tcp 10.132.0.57:8443: connect: connection refused" Jul 24 17:40:09.996770 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:09.996745 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:10.009476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.009399 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:40:10.091597 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.091565 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-openshift-service-ca\") pod \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " Jul 24 17:40:10.091597 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.091600 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls\") pod \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " Jul 24 17:40:10.091890 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.091640 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmshh\" (UniqueName: \"kubernetes.io/projected/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-kube-api-access-nmshh\") pod \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\" (UID: \"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d\") " Jul 24 17:40:10.092088 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.092045 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" (UID: "40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:40:10.094315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.094269 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-kube-api-access-nmshh" (OuterVolumeSpecName: "kube-api-access-nmshh") pod "40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" (UID: "40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d"). InnerVolumeSpecName "kube-api-access-nmshh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:40:10.094494 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.094424 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" (UID: "40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:40:10.193518 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.193483 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:10.193518 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.193517 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:10.193718 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.193531 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmshh\" (UniqueName: \"kubernetes.io/projected/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d-kube-api-access-nmshh\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:10.379612 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.379516 2552 generic.go:356] "Generic (PLEG): container finished" podID="40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" containerID="9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef" exitCode=0 Jul 24 17:40:10.379612 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.379566 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" Jul 24 17:40:10.379797 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.379615 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" event={"ID":"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d","Type":"ContainerDied","Data":"9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef"} Jul 24 17:40:10.379797 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.379673 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t" event={"ID":"40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d","Type":"ContainerDied","Data":"ee6a64b59b4ff00e3f640004ac0381b629a26fdfba4ddfa1e45c4091ec738f90"} Jul 24 17:40:10.379797 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.379696 2552 scope.go:117] "RemoveContainer" containerID="9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef" Jul 24 17:40:10.389424 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.389383 2552 scope.go:117] "RemoveContainer" containerID="9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef" Jul 24 17:40:10.389691 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:40:10.389672 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef\": container with ID starting with 9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef not found: ID does not exist" containerID="9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef" Jul 24 17:40:10.389741 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.389702 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef"} err="failed to get container status \"9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef\": rpc error: code = NotFound desc = could not find container \"9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef\": container with ID starting with 9b2e02df393fbe11223615e52376bc8053d6eff9a6dcda25d4b22a39f3bb26ef not found: ID does not exist" Jul 24 17:40:10.401609 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.401582 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t"] Jul 24 17:40:10.405486 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.405459 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-0d8e36-786864c78d-mzm6t"] Jul 24 17:40:10.964465 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:10.964437 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" path="/var/lib/kubelet/pods/40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d/volumes" Jul 24 17:40:12.759136 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:12.759092 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:14.300578 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:14.300533 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:14.701343 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:14.701309 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:21.083908 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:21.083864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:21.195002 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:21.194965 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:21.395328 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:21.395245 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:21.605172 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:21.605132 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:22.597373 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:22.597342 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:22.697768 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:22.697732 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:32.312209 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.312175 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp"] Jul 24 17:40:32.312674 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.312445 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" podUID="db7729d9-e14c-4010-8420-c6080916b1a9" containerName="maas-api" containerID="cri-o://64ea9e97b0dceb8004157890e43a49799d3d45e95d181de21f29b6ba4356a6bc" gracePeriod=30 Jul 24 17:40:32.343368 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.343315 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:32.448100 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.448052 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:32.496077 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.496050 2552 generic.go:356] "Generic (PLEG): container finished" podID="db7729d9-e14c-4010-8420-c6080916b1a9" containerID="64ea9e97b0dceb8004157890e43a49799d3d45e95d181de21f29b6ba4356a6bc" exitCode=0 Jul 24 17:40:32.496229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.496116 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" event={"ID":"db7729d9-e14c-4010-8420-c6080916b1a9","Type":"ContainerDied","Data":"64ea9e97b0dceb8004157890e43a49799d3d45e95d181de21f29b6ba4356a6bc"} Jul 24 17:40:32.545438 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.545394 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:32.591670 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.591351 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:40:32.703188 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.703152 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7729d9-e14c-4010-8420-c6080916b1a9-openshift-service-ca\") pod \"db7729d9-e14c-4010-8420-c6080916b1a9\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " Jul 24 17:40:32.703188 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.703187 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls\") pod \"db7729d9-e14c-4010-8420-c6080916b1a9\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " Jul 24 17:40:32.703487 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.703303 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp68v\" (UniqueName: \"kubernetes.io/projected/db7729d9-e14c-4010-8420-c6080916b1a9-kube-api-access-fp68v\") pod \"db7729d9-e14c-4010-8420-c6080916b1a9\" (UID: \"db7729d9-e14c-4010-8420-c6080916b1a9\") " Jul 24 17:40:32.703553 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.703522 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db7729d9-e14c-4010-8420-c6080916b1a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "db7729d9-e14c-4010-8420-c6080916b1a9" (UID: "db7729d9-e14c-4010-8420-c6080916b1a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:40:32.703700 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.703681 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db7729d9-e14c-4010-8420-c6080916b1a9-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:32.705276 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.705251 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db7729d9-e14c-4010-8420-c6080916b1a9-kube-api-access-fp68v" (OuterVolumeSpecName: "kube-api-access-fp68v") pod "db7729d9-e14c-4010-8420-c6080916b1a9" (UID: "db7729d9-e14c-4010-8420-c6080916b1a9"). InnerVolumeSpecName "kube-api-access-fp68v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:40:32.705386 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.705358 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "db7729d9-e14c-4010-8420-c6080916b1a9" (UID: "db7729d9-e14c-4010-8420-c6080916b1a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:40:32.804795 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.804760 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db7729d9-e14c-4010-8420-c6080916b1a9-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:32.804795 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:32.804796 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp68v\" (UniqueName: \"kubernetes.io/projected/db7729d9-e14c-4010-8420-c6080916b1a9-kube-api-access-fp68v\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:33.502810 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:33.502768 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" event={"ID":"db7729d9-e14c-4010-8420-c6080916b1a9","Type":"ContainerDied","Data":"c03c4ba383285dd3f06462165f2d7dfca1f22aeee05cd513ee557b9002ff56b1"} Jul 24 17:40:33.502810 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:33.502799 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp" Jul 24 17:40:33.503288 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:33.502846 2552 scope.go:117] "RemoveContainer" containerID="64ea9e97b0dceb8004157890e43a49799d3d45e95d181de21f29b6ba4356a6bc" Jul 24 17:40:33.539764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:33.539733 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp"] Jul 24 17:40:33.551526 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:33.550781 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-746c88-ff9c699-4d6mp"] Jul 24 17:40:34.964618 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:34.964585 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db7729d9-e14c-4010-8420-c6080916b1a9" path="/var/lib/kubelet/pods/db7729d9-e14c-4010-8420-c6080916b1a9/volumes" Jul 24 17:40:36.803958 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:36.803923 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:37.192566 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:37.192531 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:43.271569 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:43.271515 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:43.369852 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:43.369817 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:43.513727 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:43.513696 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7"] Jul 24 17:40:43.513926 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:43.513904 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" containerID="cri-o://a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286" gracePeriod=30 Jul 24 17:40:43.533053 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:43.532968 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:43.774134 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:43.774097 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:44.137898 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:44.137864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:44.474752 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:44.474717 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:44.787844 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:44.787753 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:47.723208 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.723172 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:47.832631 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.832589 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:47.841012 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.840983 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk"] Jul 24 17:40:47.841637 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.841620 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" containerName="maas-api" Jul 24 17:40:47.841701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.841639 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" containerName="maas-api" Jul 24 17:40:47.841701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.841662 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db7729d9-e14c-4010-8420-c6080916b1a9" containerName="maas-api" Jul 24 17:40:47.841701 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.841670 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="db7729d9-e14c-4010-8420-c6080916b1a9" containerName="maas-api" Jul 24 17:40:47.841809 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.841760 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="db7729d9-e14c-4010-8420-c6080916b1a9" containerName="maas-api" Jul 24 17:40:47.841809 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.841774 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="40b06ed8-93d1-4537-bdbb-6dfdbf40ba1d" containerName="maas-api" Jul 24 17:40:47.845304 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.845286 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:47.847529 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.847505 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-api-a-bb341a\"" Jul 24 17:40:47.855981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.855955 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk"] Jul 24 17:40:47.923594 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.923553 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:47.948023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.947987 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdt8g\" (UniqueName: \"kubernetes.io/projected/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-kube-api-access-pdt8g\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:47.948205 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.948029 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:47.948205 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:47.948122 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.049644 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.049552 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdt8g\" (UniqueName: \"kubernetes.io/projected/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-kube-api-access-pdt8g\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.049644 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.049593 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.049883 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.049646 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.049942 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:40:48.049880 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-api-a-bb341a: secret "maas-api-serving-cert-e2e-api-a-bb341a" not found Jul 24 17:40:48.050001 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:40:48.049949 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls podName:e26df5e9-e92d-4c88-b9bd-2b6c68403fe5 nodeName:}" failed. No retries permitted until 2026-07-24 17:40:48.549931051 +0000 UTC m=+2266.048333575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls") pod "maas-api-e2e-api-a-bb341a-698b467766-n7lsk" (UID: "e26df5e9-e92d-4c88-b9bd-2b6c68403fe5") : secret "maas-api-serving-cert-e2e-api-a-bb341a" not found Jul 24 17:40:48.050337 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.050313 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-openshift-service-ca\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.058212 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.058188 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdt8g\" (UniqueName: \"kubernetes.io/projected/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-kube-api-access-pdt8g\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.194570 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.194531 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:48.409514 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.409484 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:48.503525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.503490 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:48.554847 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.554818 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.557235 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.557215 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls\") pod \"maas-api-e2e-api-a-bb341a-698b467766-n7lsk\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.769613 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.769584 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:48.786574 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.786552 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:40:48.912211 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.912178 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk"] Jul 24 17:40:48.914749 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:40:48.914719 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26df5e9_e92d_4c88_b9bd_2b6c68403fe5.slice/crio-3b0740df5d77d1405686fea90931b9f5e8a65b07efaa58d295a6fde87b1bd1b6 WatchSource:0}: Error finding container 3b0740df5d77d1405686fea90931b9f5e8a65b07efaa58d295a6fde87b1bd1b6: Status 404 returned error can't find the container with id 3b0740df5d77d1405686fea90931b9f5e8a65b07efaa58d295a6fde87b1bd1b6 Jul 24 17:40:48.916024 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.916008 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:40:48.959774 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959705 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-data\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.959774 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959734 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e606d-646a-4908-af06-c95941cdc2df-istiod-ca-cert\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.959774 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959752 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-credential-socket\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.960029 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959850 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-socket\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.960029 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959879 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tbgw\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-kube-api-access-9tbgw\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.960029 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959901 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-istio-token\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.960029 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959947 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-envoy\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.960029 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.959979 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-certs\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.960029 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960008 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e606d-646a-4908-af06-c95941cdc2df-istio-podinfo\") pod \"f23e606d-646a-4908-af06-c95941cdc2df\" (UID: \"f23e606d-646a-4908-af06-c95941cdc2df\") " Jul 24 17:40:48.960330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960066 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:40:48.960330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960142 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:40:48.960330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960192 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23e606d-646a-4908-af06-c95941cdc2df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:40:48.960522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960376 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:48.960522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960397 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e606d-646a-4908-af06-c95941cdc2df-istiod-ca-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:48.960522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960444 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-credential-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:48.960522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960442 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:40:48.960761 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.960740 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-data" (OuterVolumeSpecName: "istio-data") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:40:48.962307 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.962282 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-kube-api-access-9tbgw" (OuterVolumeSpecName: "kube-api-access-9tbgw") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "kube-api-access-9tbgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:40:48.962612 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.962590 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f23e606d-646a-4908-af06-c95941cdc2df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:40:48.962722 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.962692 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-istio-token" (OuterVolumeSpecName: "istio-token") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:40:48.962833 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:48.962746 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f23e606d-646a-4908-af06-c95941cdc2df" (UID: "f23e606d-646a-4908-af06-c95941cdc2df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:40:49.061213 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.061177 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-data\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:49.061213 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.061212 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbgw\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-kube-api-access-9tbgw\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:49.061401 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.061225 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e606d-646a-4908-af06-c95941cdc2df-istio-token\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:49.061401 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.061235 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-istio-envoy\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:49.061401 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.061243 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e606d-646a-4908-af06-c95941cdc2df-workload-certs\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:49.061401 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.061251 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e606d-646a-4908-af06-c95941cdc2df-istio-podinfo\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:40:49.585494 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.585453 2552 generic.go:356] "Generic (PLEG): container finished" podID="f23e606d-646a-4908-af06-c95941cdc2df" containerID="a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286" exitCode=0 Jul 24 17:40:49.585657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.585507 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" Jul 24 17:40:49.585657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.585531 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" event={"ID":"f23e606d-646a-4908-af06-c95941cdc2df","Type":"ContainerDied","Data":"a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286"} Jul 24 17:40:49.585657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.585581 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7" event={"ID":"f23e606d-646a-4908-af06-c95941cdc2df","Type":"ContainerDied","Data":"979a791519562a2bc32fc9cbcc683c0c9a0dc4cb601a0708ab962033e49bec83"} Jul 24 17:40:49.585657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.585603 2552 scope.go:117] "RemoveContainer" containerID="a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286" Jul 24 17:40:49.587371 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.587349 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" event={"ID":"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5","Type":"ContainerStarted","Data":"71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea"} Jul 24 17:40:49.587601 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.587385 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" event={"ID":"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5","Type":"ContainerStarted","Data":"3b0740df5d77d1405686fea90931b9f5e8a65b07efaa58d295a6fde87b1bd1b6"} Jul 24 17:40:49.587601 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.587468 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:49.595014 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.594996 2552 scope.go:117] "RemoveContainer" containerID="d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56" Jul 24 17:40:49.603076 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.603058 2552 scope.go:117] "RemoveContainer" containerID="a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286" Jul 24 17:40:49.603331 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:40:49.603307 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286\": container with ID starting with a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286 not found: ID does not exist" containerID="a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286" Jul 24 17:40:49.603447 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.603338 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286"} err="failed to get container status \"a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286\": rpc error: code = NotFound desc = could not find container \"a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286\": container with ID starting with a4448821c04d516539f418e6174c0c715af631807211c6ef997b559a928ba286 not found: ID does not exist" Jul 24 17:40:49.603447 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.603355 2552 scope.go:117] "RemoveContainer" containerID="d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56" Jul 24 17:40:49.603616 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:40:49.603601 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56\": container with ID starting with d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56 not found: ID does not exist" containerID="d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56" Jul 24 17:40:49.603666 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.603619 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56"} err="failed to get container status \"d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56\": rpc error: code = NotFound desc = could not find container \"d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56\": container with ID starting with d1241d3bb29475b58b1e6d0da054a3fbfcf74ad920ea1be530eca187c4c0eb56 not found: ID does not exist" Jul 24 17:40:49.614162 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.614124 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" podStartSLOduration=2.129316733 podStartE2EDuration="2.614111577s" totalImagesPullingTime="484.794844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:40:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:49.610613172 +0000 UTC m=+2267.109015719" watchObservedRunningTime="2026-07-24 17:40:49.614111577 +0000 UTC m=+2267.112514123" Jul 24 17:40:49.633081 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.633048 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7"] Jul 24 17:40:49.639171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:49.639147 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-746c88-openshift-default-6ff58d5688-44gs7"] Jul 24 17:40:50.964876 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:50.964844 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f23e606d-646a-4908-af06-c95941cdc2df" path="/var/lib/kubelet/pods/f23e606d-646a-4908-af06-c95941cdc2df/volumes" Jul 24 17:40:52.247668 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:52.247635 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:52.756476 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:52.756438 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:53.038682 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.038594 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk"] Jul 24 17:40:53.039230 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.039209 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" Jul 24 17:40:53.039279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.039235 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" Jul 24 17:40:53.039384 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.039371 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" Jul 24 17:40:53.039438 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.039391 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" Jul 24 17:40:53.044443 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.044422 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.050049 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.050031 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-bcb261-openshift-default-dockercfg-bjxc5\"" Jul 24 17:40:53.054815 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.054791 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk"] Jul 24 17:40:53.070318 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.070293 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:53.097633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097601 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-podinfo\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.097757 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097645 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-credential-socket\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.097757 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097680 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-envoy\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.097757 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097734 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-token\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.097884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097760 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-socket\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.097884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097782 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-data\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.097884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097805 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istiod-ca-cert\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.097884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097833 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp8np\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-kube-api-access-zp8np\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.098020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.097886 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-certs\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198372 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-token\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198431 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198433 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-socket\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198548 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-data\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198592 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istiod-ca-cert\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198626 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp8np\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-kube-api-access-zp8np\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198856 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198694 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-certs\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198856 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198781 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-podinfo\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198856 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198807 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-socket\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198856 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198817 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-credential-socket\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.198856 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198836 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-data\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.199111 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.198867 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-envoy\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.199111 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.199074 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-certs\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.199229 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.199148 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-credential-socket\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.199352 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.199330 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istiod-ca-cert\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.201025 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.200992 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-envoy\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.201268 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.201247 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-podinfo\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.206190 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.206162 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-token\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.206391 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.206373 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp8np\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-kube-api-access-zp8np\") pod \"e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.356207 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.356129 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:53.406707 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.406558 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:53.498302 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.498276 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk"] Jul 24 17:40:53.500082 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:40:53.500058 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d3acd2_f4c0_475c_8ff5_f5d007f02fae.slice/crio-dcaf4abe7f776e5a3c19852dc7132305de4922fc5e099fc559594f9d335ecd80 WatchSource:0}: Error finding container dcaf4abe7f776e5a3c19852dc7132305de4922fc5e099fc559594f9d335ecd80: Status 404 returned error can't find the container with id dcaf4abe7f776e5a3c19852dc7132305de4922fc5e099fc559594f9d335ecd80 Jul 24 17:40:53.503848 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.503818 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:40:53.503944 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.503879 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:40:53.503944 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.503908 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:40:53.611062 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.610981 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" event={"ID":"82d3acd2-f4c0-475c-8ff5-f5d007f02fae","Type":"ContainerStarted","Data":"353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51"} Jul 24 17:40:53.611062 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.611037 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" event={"ID":"82d3acd2-f4c0-475c-8ff5-f5d007f02fae","Type":"ContainerStarted","Data":"dcaf4abe7f776e5a3c19852dc7132305de4922fc5e099fc559594f9d335ecd80"} Jul 24 17:40:53.632470 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:53.632400 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podStartSLOduration=0.632385001 podStartE2EDuration="632.385001ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:53.630570885 +0000 UTC m=+2271.128973445" watchObservedRunningTime="2026-07-24 17:40:53.632385001 +0000 UTC m=+2271.130787546" Jul 24 17:40:54.356584 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:54.356540 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:54.361511 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:54.361484 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:54.616361 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:54.616277 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:54.617304 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:54.617283 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:40:55.594227 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.594198 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:55.599069 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.599046 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:40:55.676073 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.676034 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk"] Jul 24 17:40:55.676734 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.676710 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" Jul 24 17:40:55.676872 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.676742 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f23e606d-646a-4908-af06-c95941cdc2df" containerName="istio-proxy" Jul 24 17:40:55.676872 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.676707 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:55.680606 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.680583 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.683027 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.683009 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-api-b-bcb261\"" Jul 24 17:40:55.689819 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.689658 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk"] Jul 24 17:40:55.720946 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.720918 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.721087 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.720979 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.721087 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.721056 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wq9\" (UniqueName: \"kubernetes.io/projected/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-kube-api-access-v7wq9\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.747702 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.747675 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:55.822131 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.822098 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.822131 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.822134 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.822392 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.822185 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wq9\" (UniqueName: \"kubernetes.io/projected/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-kube-api-access-v7wq9\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.822392 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:40:55.822237 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-api-b-bcb261: secret "maas-api-serving-cert-e2e-api-b-bcb261" not found Jul 24 17:40:55.822392 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:40:55.822305 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls podName:3ef9a0f9-9838-44d1-a019-7e55f429c2bc nodeName:}" failed. No retries permitted until 2026-07-24 17:40:56.322288481 +0000 UTC m=+2273.820691007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls") pod "maas-api-e2e-api-b-bcb261-798487474-8dqdk" (UID: "3ef9a0f9-9838-44d1-a019-7e55f429c2bc") : secret "maas-api-serving-cert-e2e-api-b-bcb261" not found Jul 24 17:40:55.823064 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.823042 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-openshift-service-ca\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.830899 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.830873 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wq9\" (UniqueName: \"kubernetes.io/projected/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-kube-api-access-v7wq9\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:55.953952 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:55.953918 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:56.244370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:56.244291 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:56.327076 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:56.327045 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:56.329424 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:56.329382 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls\") pod \"maas-api-e2e-api-b-bcb261-798487474-8dqdk\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:56.347744 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:56.347718 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:40:56.595304 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:56.595208 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:56.742224 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:56.742194 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk"] Jul 24 17:40:56.742681 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:40:56.742653 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef9a0f9_9838_44d1_a019_7e55f429c2bc.slice/crio-cbdcb8f0ad4d537b7044cdb7f7a2f1a5c53d15d7e7e1273ce395f054c54becfc WatchSource:0}: Error finding container cbdcb8f0ad4d537b7044cdb7f7a2f1a5c53d15d7e7e1273ce395f054c54becfc: Status 404 returned error can't find the container with id cbdcb8f0ad4d537b7044cdb7f7a2f1a5c53d15d7e7e1273ce395f054c54becfc Jul 24 17:40:57.635201 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:57.635165 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" event={"ID":"3ef9a0f9-9838-44d1-a019-7e55f429c2bc","Type":"ContainerStarted","Data":"7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f"} Jul 24 17:40:57.635601 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:57.635226 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" event={"ID":"3ef9a0f9-9838-44d1-a019-7e55f429c2bc","Type":"ContainerStarted","Data":"cbdcb8f0ad4d537b7044cdb7f7a2f1a5c53d15d7e7e1273ce395f054c54becfc"} Jul 24 17:40:57.635601 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:57.635255 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:40:57.652875 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:40:57.652833 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" podStartSLOduration=2.338413698 podStartE2EDuration="2.652819329s" totalImagesPullingTime="314.405631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:40:57.652028585 +0000 UTC m=+2275.150431124" watchObservedRunningTime="2026-07-24 17:40:57.652819329 +0000 UTC m=+2275.151221874" Jul 24 17:41:00.088231 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:00.088199 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:03.647267 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:03.647185 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:41:10.658638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:10.658570 2552 patch_prober.go:28] interesting pod/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 17:41:10.659184 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:10.658707 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 17:41:11.712102 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:11.712069 2552 generic.go:356] "Generic (PLEG): container finished" podID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerID="353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51" exitCode=0 Jul 24 17:41:11.712666 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:11.712142 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" event={"ID":"82d3acd2-f4c0-475c-8ff5-f5d007f02fae","Type":"ContainerDied","Data":"353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51"} Jul 24 17:41:11.712738 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:11.712690 2552 scope.go:117] "RemoveContainer" containerID="353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51" Jul 24 17:41:11.713751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:11.713726 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:41:11.713805 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:11.713794 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:41:11.713850 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:11.713826 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:41:12.721082 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:12.720104 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" event={"ID":"82d3acd2-f4c0-475c-8ff5-f5d007f02fae","Type":"ContainerStarted","Data":"f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52"} Jul 24 17:41:13.356626 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:13.356592 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:41:13.356818 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:13.356798 2552 patch_prober.go:28] interesting pod/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 24 17:41:13.356879 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:13.356844 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 24 17:41:14.356810 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:14.356775 2552 patch_prober.go:28] interesting pod/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 24 17:41:14.357216 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:14.356840 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 24 17:41:15.356927 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:15.356892 2552 patch_prober.go:28] interesting pod/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 24 17:41:15.357383 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:15.356948 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 24 17:41:16.357528 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:16.357486 2552 patch_prober.go:28] interesting pod/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Jul 24 17:41:16.358091 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:16.357545 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Jul 24 17:41:17.360206 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:17.360173 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:41:17.360624 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:17.360437 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:41:17.361137 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:17.361121 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:41:23.900362 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:23.900327 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk"] Jul 24 17:41:23.900859 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:23.900590 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" podUID="e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" containerName="maas-api" containerID="cri-o://71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea" gracePeriod=30 Jul 24 17:41:23.927712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:23.927688 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:24.028066 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.027993 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:24.127977 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.127938 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:24.166633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.166609 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:41:24.291122 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.291090 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdt8g\" (UniqueName: \"kubernetes.io/projected/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-kube-api-access-pdt8g\") pod \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " Jul 24 17:41:24.291287 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.291134 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls\") pod \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " Jul 24 17:41:24.291287 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.291166 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-openshift-service-ca\") pod \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\" (UID: \"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5\") " Jul 24 17:41:24.291636 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.291612 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" (UID: "e26df5e9-e92d-4c88-b9bd-2b6c68403fe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:41:24.293234 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.293205 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" (UID: "e26df5e9-e92d-4c88-b9bd-2b6c68403fe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:41:24.293234 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.293220 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-kube-api-access-pdt8g" (OuterVolumeSpecName: "kube-api-access-pdt8g") pod "e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" (UID: "e26df5e9-e92d-4c88-b9bd-2b6c68403fe5"). InnerVolumeSpecName "kube-api-access-pdt8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:41:24.392128 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.392097 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdt8g\" (UniqueName: \"kubernetes.io/projected/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-kube-api-access-pdt8g\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:41:24.392128 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.392123 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:41:24.392128 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.392134 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:41:24.790920 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.790889 2552 generic.go:356] "Generic (PLEG): container finished" podID="e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" containerID="71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea" exitCode=0 Jul 24 17:41:24.791097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.790935 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" Jul 24 17:41:24.791097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.790970 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" event={"ID":"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5","Type":"ContainerDied","Data":"71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea"} Jul 24 17:41:24.791097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.791021 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk" event={"ID":"e26df5e9-e92d-4c88-b9bd-2b6c68403fe5","Type":"ContainerDied","Data":"3b0740df5d77d1405686fea90931b9f5e8a65b07efaa58d295a6fde87b1bd1b6"} Jul 24 17:41:24.791097 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.791043 2552 scope.go:117] "RemoveContainer" containerID="71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea" Jul 24 17:41:24.800323 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.800308 2552 scope.go:117] "RemoveContainer" containerID="71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea" Jul 24 17:41:24.800593 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:41:24.800575 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea\": container with ID starting with 71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea not found: ID does not exist" containerID="71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea" Jul 24 17:41:24.800650 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.800601 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea"} err="failed to get container status \"71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea\": rpc error: code = NotFound desc = could not find container \"71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea\": container with ID starting with 71a75a8c7a97735c0208c3ac44c756c5be00e55663d4a98c4d00d87391247cea not found: ID does not exist" Jul 24 17:41:24.812617 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.812597 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk"] Jul 24 17:41:24.816464 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.816443 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bb341a-698b467766-n7lsk"] Jul 24 17:41:24.966518 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:24.966482 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" path="/var/lib/kubelet/pods/e26df5e9-e92d-4c88-b9bd-2b6c68403fe5/volumes" Jul 24 17:41:27.053571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:27.053538 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:28.322673 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:28.322632 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:28.689026 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:28.688948 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:34.732199 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:34.732154 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:35.919106 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:35.919038 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:50.989802 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:50.989760 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk"] Jul 24 17:41:50.990337 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:50.990123 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" podUID="3ef9a0f9-9838-44d1-a019-7e55f429c2bc" containerName="maas-api" containerID="cri-o://7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f" gracePeriod=30 Jul 24 17:41:51.025680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.025652 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:51.120890 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.120853 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:51.225441 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.224813 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:51.255520 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.255452 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:41:51.350366 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.350335 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls\") pod \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " Jul 24 17:41:51.350549 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.350389 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-openshift-service-ca\") pod \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " Jul 24 17:41:51.350549 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.350464 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7wq9\" (UniqueName: \"kubernetes.io/projected/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-kube-api-access-v7wq9\") pod \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\" (UID: \"3ef9a0f9-9838-44d1-a019-7e55f429c2bc\") " Jul 24 17:41:51.350695 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.350667 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3ef9a0f9-9838-44d1-a019-7e55f429c2bc" (UID: "3ef9a0f9-9838-44d1-a019-7e55f429c2bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:41:51.350791 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.350771 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:41:51.352493 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.352467 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3ef9a0f9-9838-44d1-a019-7e55f429c2bc" (UID: "3ef9a0f9-9838-44d1-a019-7e55f429c2bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:41:51.352602 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.352470 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-kube-api-access-v7wq9" (OuterVolumeSpecName: "kube-api-access-v7wq9") pod "3ef9a0f9-9838-44d1-a019-7e55f429c2bc" (UID: "3ef9a0f9-9838-44d1-a019-7e55f429c2bc"). InnerVolumeSpecName "kube-api-access-v7wq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:41:51.451500 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.451469 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:41:51.451500 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.451497 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wq9\" (UniqueName: \"kubernetes.io/projected/3ef9a0f9-9838-44d1-a019-7e55f429c2bc-kube-api-access-v7wq9\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:41:51.926806 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.926775 2552 generic.go:356] "Generic (PLEG): container finished" podID="3ef9a0f9-9838-44d1-a019-7e55f429c2bc" containerID="7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f" exitCode=0 Jul 24 17:41:51.926973 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.926818 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" Jul 24 17:41:51.926973 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.926860 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" event={"ID":"3ef9a0f9-9838-44d1-a019-7e55f429c2bc","Type":"ContainerDied","Data":"7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f"} Jul 24 17:41:51.926973 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.926918 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk" event={"ID":"3ef9a0f9-9838-44d1-a019-7e55f429c2bc","Type":"ContainerDied","Data":"cbdcb8f0ad4d537b7044cdb7f7a2f1a5c53d15d7e7e1273ce395f054c54becfc"} Jul 24 17:41:51.926973 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.926946 2552 scope.go:117] "RemoveContainer" containerID="7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f" Jul 24 17:41:51.936500 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.936479 2552 scope.go:117] "RemoveContainer" containerID="7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f" Jul 24 17:41:51.936773 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:41:51.936754 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f\": container with ID starting with 7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f not found: ID does not exist" containerID="7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f" Jul 24 17:41:51.936826 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.936781 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f"} err="failed to get container status \"7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f\": rpc error: code = NotFound desc = could not find container \"7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f\": container with ID starting with 7b4713d269d285f5664f77c79010cf4df282e4ae35a4925dd8ef4067914e9a6f not found: ID does not exist" Jul 24 17:41:51.949630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.949609 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk"] Jul 24 17:41:51.956995 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:51.956975 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bcb261-798487474-8dqdk"] Jul 24 17:41:52.677270 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:52.677238 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:52.964309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:52.964226 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef9a0f9-9838-44d1-a019-7e55f429c2bc" path="/var/lib/kubelet/pods/3ef9a0f9-9838-44d1-a019-7e55f429c2bc/volumes" Jul 24 17:41:55.462614 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:55.462527 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:41:55.862700 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:41:55.862617 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:01.852893 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:01.852860 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:01.973600 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:01.973569 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:02.159327 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:02.159288 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk"] Jul 24 17:42:02.159631 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:02.159589 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" containerID="cri-o://f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52" gracePeriod=30 Jul 24 17:42:02.182979 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:02.182945 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:02.363072 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:02.363029 2552 patch_prober.go:28] interesting pod/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 17:42:02.363244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:02.363124 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:42:02.405574 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:02.405546 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:02.776442 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:02.776394 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:03.081183 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:03.081076 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:03.476628 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:03.476580 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:06.085670 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.085633 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:06.183983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.183949 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:06.324113 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.324073 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5"] Jul 24 17:42:06.324764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.324739 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" containerName="maas-api" Jul 24 17:42:06.324764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.324763 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" containerName="maas-api" Jul 24 17:42:06.324981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.324774 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef9a0f9-9838-44d1-a019-7e55f429c2bc" containerName="maas-api" Jul 24 17:42:06.324981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.324783 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef9a0f9-9838-44d1-a019-7e55f429c2bc" containerName="maas-api" Jul 24 17:42:06.324981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.324905 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26df5e9-e92d-4c88-b9bd-2b6c68403fe5" containerName="maas-api" Jul 24 17:42:06.324981 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.324920 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef9a0f9-9838-44d1-a019-7e55f429c2bc" containerName="maas-api" Jul 24 17:42:06.329782 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.329761 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.334031 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.334011 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-shared-a-028d4b\"" Jul 24 17:42:06.343060 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.343035 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5"] Jul 24 17:42:06.384370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.384335 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268af474-9c86-4f56-9e19-8e6fef98b8c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.384577 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.384386 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sbrq\" (UniqueName: \"kubernetes.io/projected/268af474-9c86-4f56-9e19-8e6fef98b8c3-kube-api-access-6sbrq\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.384577 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.384497 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.423037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.422341 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:06.485367 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.485334 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.485573 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.485446 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268af474-9c86-4f56-9e19-8e6fef98b8c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.485573 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.485478 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sbrq\" (UniqueName: \"kubernetes.io/projected/268af474-9c86-4f56-9e19-8e6fef98b8c3-kube-api-access-6sbrq\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.485573 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:42:06.485532 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-shared-a-028d4b: secret "maas-api-serving-cert-e2e-shared-a-028d4b" not found Jul 24 17:42:06.485749 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:42:06.485613 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls podName:268af474-9c86-4f56-9e19-8e6fef98b8c3 nodeName:}" failed. No retries permitted until 2026-07-24 17:42:06.985592025 +0000 UTC m=+2344.483994556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls") pod "maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" (UID: "268af474-9c86-4f56-9e19-8e6fef98b8c3") : secret "maas-api-serving-cert-e2e-shared-a-028d4b" not found Jul 24 17:42:06.486102 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.486085 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268af474-9c86-4f56-9e19-8e6fef98b8c3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.506219 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.506190 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sbrq\" (UniqueName: \"kubernetes.io/projected/268af474-9c86-4f56-9e19-8e6fef98b8c3-kube-api-access-6sbrq\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.668019 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.667990 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:06.947385 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.947304 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:06.990195 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.990166 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:06.992725 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:06.992702 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls\") pod \"maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:07.045874 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.045839 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:07.239573 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.239492 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:07.441316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.441286 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:42:07.597112 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597085 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-token\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.597182 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597166 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp8np\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-kube-api-access-zp8np\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.597235 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597198 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istiod-ca-cert\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.597294 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597266 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-podinfo\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.597359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597100 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5"] Jul 24 17:42:07.597359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597303 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-data\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.597359 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597343 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-credential-socket\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.597715 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597652 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:42:07.597715 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597654 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:42:07.597992 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.597965 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-certs\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.598121 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.598013 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-socket\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.598121 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.598046 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-envoy\") pod \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\" (UID: \"82d3acd2-f4c0-475c-8ff5-f5d007f02fae\") " Jul 24 17:42:07.598722 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.598453 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-credential-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.598722 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.598477 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istiod-ca-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.598722 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.598686 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:42:07.598935 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.598842 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-data" (OuterVolumeSpecName: "istio-data") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:42:07.599093 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.599042 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:42:07.599707 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.599682 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-token" (OuterVolumeSpecName: "istio-token") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:42:07.599906 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.599883 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-kube-api-access-zp8np" (OuterVolumeSpecName: "kube-api-access-zp8np") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "kube-api-access-zp8np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:42:07.599998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.599885 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:42:07.600052 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:42:07.599992 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268af474_9c86_4f56_9e19_8e6fef98b8c3.slice/crio-69dde35aa00affb4650f093c2d59316e235f752d2e0206396b75b0e062df1ba9 WatchSource:0}: Error finding container 69dde35aa00affb4650f093c2d59316e235f752d2e0206396b75b0e062df1ba9: Status 404 returned error can't find the container with id 69dde35aa00affb4650f093c2d59316e235f752d2e0206396b75b0e062df1ba9 Jul 24 17:42:07.600633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.600614 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82d3acd2-f4c0-475c-8ff5-f5d007f02fae" (UID: "82d3acd2-f4c0-475c-8ff5-f5d007f02fae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:42:07.699460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.699392 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-data\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.699460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.699448 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-certs\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.699460 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.699462 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-workload-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.699705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.699474 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-envoy\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.699705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.699486 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-token\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.699705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.699499 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp8np\" (UniqueName: \"kubernetes.io/projected/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-kube-api-access-zp8np\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:07.699705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:07.699512 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82d3acd2-f4c0-475c-8ff5-f5d007f02fae-istio-podinfo\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:42:08.009898 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.009864 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" event={"ID":"268af474-9c86-4f56-9e19-8e6fef98b8c3","Type":"ContainerStarted","Data":"69dde35aa00affb4650f093c2d59316e235f752d2e0206396b75b0e062df1ba9"} Jul 24 17:42:08.011377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.011353 2552 generic.go:356] "Generic (PLEG): container finished" podID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerID="f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52" exitCode=0 Jul 24 17:42:08.011498 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.011404 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" Jul 24 17:42:08.011498 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.011443 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" event={"ID":"82d3acd2-f4c0-475c-8ff5-f5d007f02fae","Type":"ContainerDied","Data":"f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52"} Jul 24 17:42:08.011574 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.011500 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk" event={"ID":"82d3acd2-f4c0-475c-8ff5-f5d007f02fae","Type":"ContainerDied","Data":"dcaf4abe7f776e5a3c19852dc7132305de4922fc5e099fc559594f9d335ecd80"} Jul 24 17:42:08.011574 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.011531 2552 scope.go:117] "RemoveContainer" containerID="f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52" Jul 24 17:42:08.021621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.021598 2552 scope.go:117] "RemoveContainer" containerID="353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51" Jul 24 17:42:08.030895 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.030876 2552 scope.go:117] "RemoveContainer" containerID="f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52" Jul 24 17:42:08.031166 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:42:08.031147 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52\": container with ID starting with f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52 not found: ID does not exist" containerID="f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52" Jul 24 17:42:08.031208 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.031175 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52"} err="failed to get container status \"f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52\": rpc error: code = NotFound desc = could not find container \"f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52\": container with ID starting with f8591c31fde1d786375e5c2ff9f8759747e2d0b05c9f9243a4d76be14b50ba52 not found: ID does not exist" Jul 24 17:42:08.031208 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.031192 2552 scope.go:117] "RemoveContainer" containerID="353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51" Jul 24 17:42:08.031466 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:42:08.031439 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51\": container with ID starting with 353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51 not found: ID does not exist" containerID="353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51" Jul 24 17:42:08.031576 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.031470 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51"} err="failed to get container status \"353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51\": rpc error: code = NotFound desc = could not find container \"353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51\": container with ID starting with 353b78382b69ccb68c41a29ef0b57ffc6f33ab850f9aa6f8bfcb495076c6aa51 not found: ID does not exist" Jul 24 17:42:08.035440 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.035402 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk"] Jul 24 17:42:08.038723 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.038703 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-bcb261-openshift-default-65747976cd-lbqdk"] Jul 24 17:42:08.965303 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:08.965269 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" path="/var/lib/kubelet/pods/82d3acd2-f4c0-475c-8ff5-f5d007f02fae/volumes" Jul 24 17:42:09.017723 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:09.017678 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" event={"ID":"268af474-9c86-4f56-9e19-8e6fef98b8c3","Type":"ContainerStarted","Data":"465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9"} Jul 24 17:42:09.017933 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:09.017791 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:09.036231 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:09.036179 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" podStartSLOduration=2.15073757 podStartE2EDuration="3.036161946s" totalImagesPullingTime="885.424376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:09.03475373 +0000 UTC m=+2346.533156275" watchObservedRunningTime="2026-07-24 17:42:09.036161946 +0000 UTC m=+2346.534564492" Jul 24 17:42:10.594056 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:10.594021 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:10.935891 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:10.935858 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:11.230756 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.230668 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4"] Jul 24 17:42:11.231291 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.231275 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" Jul 24 17:42:11.231339 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.231293 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" Jul 24 17:42:11.231339 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.231308 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" Jul 24 17:42:11.231339 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.231318 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" Jul 24 17:42:11.231470 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.231403 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" Jul 24 17:42:11.231470 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.231432 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d3acd2-f4c0-475c-8ff5-f5d007f02fae" containerName="istio-proxy" Jul 24 17:42:11.235066 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.235045 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.237657 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.237638 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-a11aa7-openshift-default-dockercfg-8f7jh\"" Jul 24 17:42:11.247279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.247253 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4"] Jul 24 17:42:11.271522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.271495 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:11.332363 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332332 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdmb\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-kube-api-access-jzdmb\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332520 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332372 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-socket\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332520 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332396 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-istio-token\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332520 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332433 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24414c34-102b-416b-a4c5-0df50cad0a52-istiod-ca-cert\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332520 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332471 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-data\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332520 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332501 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-certs\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332558 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-envoy\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332617 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24414c34-102b-416b-a4c5-0df50cad0a52-istio-podinfo\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.332705 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.332650 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-credential-socket\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.394840 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.394811 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:11.433387 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433357 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-envoy\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433418 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24414c34-102b-416b-a4c5-0df50cad0a52-istio-podinfo\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433452 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-credential-socket\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433493 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdmb\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-kube-api-access-jzdmb\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433562 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433523 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-socket\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433576 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-istio-token\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433614 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24414c34-102b-416b-a4c5-0df50cad0a52-istiod-ca-cert\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433675 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-data\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.433786 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433725 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-certs\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.434020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433853 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-credential-socket\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.434020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.433900 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-socket\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.434133 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.434076 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-certs\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.434189 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.434155 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-data\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.434510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.434488 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24414c34-102b-416b-a4c5-0df50cad0a52-istiod-ca-cert\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.435682 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.435662 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-envoy\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.435980 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.435960 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24414c34-102b-416b-a4c5-0df50cad0a52-istio-podinfo\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.442681 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.442656 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-istio-token\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.442763 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.442658 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdmb\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-kube-api-access-jzdmb\") pod \"e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.546536 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.546465 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:11.684349 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.684323 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4"] Jul 24 17:42:11.685360 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:42:11.685336 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24414c34_102b_416b_a4c5_0df50cad0a52.slice/crio-8c44e08daf90c97daba88eb09df263e0150610c771486d2a7e44d22749ad8d7c WatchSource:0}: Error finding container 8c44e08daf90c97daba88eb09df263e0150610c771486d2a7e44d22749ad8d7c: Status 404 returned error can't find the container with id 8c44e08daf90c97daba88eb09df263e0150610c771486d2a7e44d22749ad8d7c Jul 24 17:42:11.687608 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.687576 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:42:11.687711 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.687635 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:42:11.687711 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:11.687663 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:42:12.037356 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:12.037317 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" event={"ID":"24414c34-102b-416b-a4c5-0df50cad0a52","Type":"ContainerStarted","Data":"1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786"} Jul 24 17:42:12.037539 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:12.037427 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" event={"ID":"24414c34-102b-416b-a4c5-0df50cad0a52","Type":"ContainerStarted","Data":"8c44e08daf90c97daba88eb09df263e0150610c771486d2a7e44d22749ad8d7c"} Jul 24 17:42:12.057185 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:12.057137 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" podStartSLOduration=1.057122852 podStartE2EDuration="1.057122852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:12.056078004 +0000 UTC m=+2349.554480551" watchObservedRunningTime="2026-07-24 17:42:12.057122852 +0000 UTC m=+2349.555525402" Jul 24 17:42:12.547330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:12.547295 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:12.552396 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:12.552371 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:13.043426 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:13.043388 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:13.044484 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:13.044464 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:13.974547 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:13.974514 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:14.076249 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.076213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:14.298574 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.298502 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz"] Jul 24 17:42:14.302324 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.302306 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.305806 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.305782 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-shared-b-a11aa7\"" Jul 24 17:42:14.335233 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.335209 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:14.336139 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.336117 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz"] Jul 24 17:42:14.461772 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.461720 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe159605-888f-48d6-a81b-0959cf4adf4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.461772 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.461791 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2r6z\" (UniqueName: \"kubernetes.io/projected/fe159605-888f-48d6-a81b-0959cf4adf4f-kube-api-access-p2r6z\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.462073 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.461847 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe159605-888f-48d6-a81b-0959cf4adf4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.542992 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.542957 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:14.563377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.563295 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe159605-888f-48d6-a81b-0959cf4adf4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.563377 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.563340 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2r6z\" (UniqueName: \"kubernetes.io/projected/fe159605-888f-48d6-a81b-0959cf4adf4f-kube-api-access-p2r6z\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.563611 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.563515 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe159605-888f-48d6-a81b-0959cf4adf4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.564010 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.563986 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe159605-888f-48d6-a81b-0959cf4adf4f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.565928 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.565911 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe159605-888f-48d6-a81b-0959cf4adf4f-maas-api-tls\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.571814 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.571784 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2r6z\" (UniqueName: \"kubernetes.io/projected/fe159605-888f-48d6-a81b-0959cf4adf4f-kube-api-access-p2r6z\") pod \"maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.613630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.613607 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:14.755829 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.755804 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz"] Jul 24 17:42:14.757565 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:42:14.757537 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe159605_888f_48d6_a81b_0959cf4adf4f.slice/crio-725b4eb6c47b8b58b62fbf92c928357c2f13a6ee076f4fa27f5aab063974eea1 WatchSource:0}: Error finding container 725b4eb6c47b8b58b62fbf92c928357c2f13a6ee076f4fa27f5aab063974eea1: Status 404 returned error can't find the container with id 725b4eb6c47b8b58b62fbf92c928357c2f13a6ee076f4fa27f5aab063974eea1 Jul 24 17:42:14.961620 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:14.961554 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:15.030145 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:15.030121 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:42:15.054344 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:15.054316 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" event={"ID":"fe159605-888f-48d6-a81b-0959cf4adf4f","Type":"ContainerStarted","Data":"725b4eb6c47b8b58b62fbf92c928357c2f13a6ee076f4fa27f5aab063974eea1"} Jul 24 17:42:15.063777 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:15.063746 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:16.061317 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:16.061278 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" event={"ID":"fe159605-888f-48d6-a81b-0959cf4adf4f","Type":"ContainerStarted","Data":"4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0"} Jul 24 17:42:16.061718 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:16.061341 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:16.078884 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:16.078840 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" podStartSLOduration=1.7643107279999999 podStartE2EDuration="2.078822798s" totalImagesPullingTime="314.51207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:16.078088565 +0000 UTC m=+2353.576491111" watchObservedRunningTime="2026-07-24 17:42:16.078822798 +0000 UTC m=+2353.577225346" Jul 24 17:42:18.467527 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:18.467482 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:22.072230 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:22.072200 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:42:28.045156 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.045120 2552 patch_prober.go:28] interesting pod/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:42:28.045825 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.045216 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 24 17:42:28.129753 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.129721 2552 generic.go:356] "Generic (PLEG): container finished" podID="24414c34-102b-416b-a4c5-0df50cad0a52" containerID="1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786" exitCode=0 Jul 24 17:42:28.129908 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.129769 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" event={"ID":"24414c34-102b-416b-a4c5-0df50cad0a52","Type":"ContainerDied","Data":"1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786"} Jul 24 17:42:28.130177 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.130162 2552 scope.go:117] "RemoveContainer" containerID="1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786" Jul 24 17:42:28.131228 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.131198 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:42:28.131297 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.131281 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:42:28.131340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:28.131326 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:42:29.143387 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:29.143338 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" event={"ID":"24414c34-102b-416b-a4c5-0df50cad0a52","Type":"ContainerStarted","Data":"406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667"} Jul 24 17:42:29.547731 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:29.547693 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:29.548070 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:29.548035 2552 patch_prober.go:28] interesting pod/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:42:29.548182 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:29.548114 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 24 17:42:30.547462 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:30.547384 2552 patch_prober.go:28] interesting pod/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:42:30.548023 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:30.547474 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 24 17:42:31.547844 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:31.547805 2552 patch_prober.go:28] interesting pod/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:42:31.548245 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:31.547874 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 24 17:42:32.547656 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:32.547615 2552 patch_prober.go:28] interesting pod/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body= Jul 24 17:42:32.547876 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:32.547673 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" Jul 24 17:42:33.551753 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:33.551722 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:33.552137 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:33.552045 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:33.552595 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:33.552575 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:42:38.649193 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:38.649146 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:42:57.171790 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:42:57.171755 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:43:03.106116 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:43:03.106085 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:43:03.116337 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:43:03.116311 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:43:04.770694 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:43:04.770656 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:45:08.625915 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:45:08.625817 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:45:09.839244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:45:09.839186 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:45:23.309716 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:45:23.309678 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:45:32.113864 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:45:32.113828 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:45:55.111002 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:45:55.110964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:45:58.937853 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:45:58.937817 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:46:37.770861 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:37.770820 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:46:39.137865 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:39.137834 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:46:46.582368 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.582326 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz"] Jul 24 17:46:46.590906 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.590879 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.594389 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.594361 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-028d4b\"/\"openshift-service-ca.crt\"" Jul 24 17:46:46.594621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.594451 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-028d4b\"/\"rate-test-model-028d4b-kserve-self-signed-certs\"" Jul 24 17:46:46.594621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.594460 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-028d4b\"/\"default-dockercfg-7p4cw\"" Jul 24 17:46:46.594621 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.594365 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-028d4b\"/\"kube-root-ca.crt\"" Jul 24 17:46:46.597634 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.597613 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz"] Jul 24 17:46:46.688983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.688940 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:46:46.782121 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.782087 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-home\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.782121 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.782121 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-tmp-dir\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.782348 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.782142 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54mn\" (UniqueName: \"kubernetes.io/projected/000c855f-9388-4163-9d9a-eae0a1dfaca0-kube-api-access-k54mn\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.782348 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.782217 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000c855f-9388-4163-9d9a-eae0a1dfaca0-tls-certs\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.782348 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.782309 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-dshm\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.782348 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.782331 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-model-cache\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.882892 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.882803 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000c855f-9388-4163-9d9a-eae0a1dfaca0-tls-certs\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.882892 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.882860 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-dshm\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.882892 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.882885 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-model-cache\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.883168 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.882966 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-home\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.883168 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.882991 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-tmp-dir\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.883168 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.883010 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54mn\" (UniqueName: \"kubernetes.io/projected/000c855f-9388-4163-9d9a-eae0a1dfaca0-kube-api-access-k54mn\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.883419 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.883382 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-tmp-dir\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.883572 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.883471 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-model-cache\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.883572 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.883507 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-home\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.885116 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.885094 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-dshm\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.885399 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.885383 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000c855f-9388-4163-9d9a-eae0a1dfaca0-tls-certs\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.892075 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.892054 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54mn\" (UniqueName: \"kubernetes.io/projected/000c855f-9388-4163-9d9a-eae0a1dfaca0-kube-api-access-k54mn\") pod \"rate-test-model-028d4b-kserve-5596996455-79kqz\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:46.901492 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:46.901471 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:47.051525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:47.051499 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz"] Jul 24 17:46:47.053611 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:46:47.053580 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod000c855f_9388_4163_9d9a_eae0a1dfaca0.slice/crio-c9b50d4f83bb68d0499601aa6cba92a18926d9ae39fe7cdede1d8bc960f4e41a WatchSource:0}: Error finding container c9b50d4f83bb68d0499601aa6cba92a18926d9ae39fe7cdede1d8bc960f4e41a: Status 404 returned error can't find the container with id c9b50d4f83bb68d0499601aa6cba92a18926d9ae39fe7cdede1d8bc960f4e41a Jul 24 17:46:47.468033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:47.467998 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" event={"ID":"000c855f-9388-4163-9d9a-eae0a1dfaca0","Type":"ContainerStarted","Data":"25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a"} Jul 24 17:46:47.468033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:47.468038 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" event={"ID":"000c855f-9388-4163-9d9a-eae0a1dfaca0","Type":"ContainerStarted","Data":"c9b50d4f83bb68d0499601aa6cba92a18926d9ae39fe7cdede1d8bc960f4e41a"} Jul 24 17:46:47.492944 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:47.492898 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" podStartSLOduration=1.49288311 podStartE2EDuration="1.49288311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:46:47.490812258 +0000 UTC m=+2624.989214804" watchObservedRunningTime="2026-07-24 17:46:47.49288311 +0000 UTC m=+2624.991285691" Jul 24 17:46:56.902132 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:56.902099 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:56.902132 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:56.902144 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:56.914454 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:56.914432 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:46:57.533040 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:46:57.533014 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:47:06.101912 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:47:06.101874 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:47:24.986524 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:47:24.986487 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:02.234102 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:02.234022 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz"] Jul 24 17:48:02.264545 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:02.264506 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:03.093203 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:03.093158 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:03.139848 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:03.139821 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:48:03.149868 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:03.149845 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:48:17.253277 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.253222 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" podUID="000c855f-9388-4163-9d9a-eae0a1dfaca0" containerName="main" containerID="cri-o://25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a" gracePeriod=45 Jul 24 17:48:17.518033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.517973 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:48:17.648337 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648304 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k54mn\" (UniqueName: \"kubernetes.io/projected/000c855f-9388-4163-9d9a-eae0a1dfaca0-kube-api-access-k54mn\") pod \"000c855f-9388-4163-9d9a-eae0a1dfaca0\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " Jul 24 17:48:17.648337 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648341 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-dshm\") pod \"000c855f-9388-4163-9d9a-eae0a1dfaca0\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " Jul 24 17:48:17.648582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648363 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-tmp-dir\") pod \"000c855f-9388-4163-9d9a-eae0a1dfaca0\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " Jul 24 17:48:17.648582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648528 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-home\") pod \"000c855f-9388-4163-9d9a-eae0a1dfaca0\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " Jul 24 17:48:17.648582 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648567 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000c855f-9388-4163-9d9a-eae0a1dfaca0-tls-certs\") pod \"000c855f-9388-4163-9d9a-eae0a1dfaca0\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " Jul 24 17:48:17.648744 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648637 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-model-cache\") pod \"000c855f-9388-4163-9d9a-eae0a1dfaca0\" (UID: \"000c855f-9388-4163-9d9a-eae0a1dfaca0\") " Jul 24 17:48:17.648744 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648637 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "000c855f-9388-4163-9d9a-eae0a1dfaca0" (UID: "000c855f-9388-4163-9d9a-eae0a1dfaca0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:17.648860 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648815 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-home" (OuterVolumeSpecName: "home") pod "000c855f-9388-4163-9d9a-eae0a1dfaca0" (UID: "000c855f-9388-4163-9d9a-eae0a1dfaca0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:17.648998 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.648965 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-model-cache" (OuterVolumeSpecName: "model-cache") pod "000c855f-9388-4163-9d9a-eae0a1dfaca0" (UID: "000c855f-9388-4163-9d9a-eae0a1dfaca0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:17.649069 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.649026 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-tmp-dir\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:48:17.649069 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.649049 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-home\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:48:17.650690 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.650663 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-dshm" (OuterVolumeSpecName: "dshm") pod "000c855f-9388-4163-9d9a-eae0a1dfaca0" (UID: "000c855f-9388-4163-9d9a-eae0a1dfaca0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:48:17.650690 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.650681 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000c855f-9388-4163-9d9a-eae0a1dfaca0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "000c855f-9388-4163-9d9a-eae0a1dfaca0" (UID: "000c855f-9388-4163-9d9a-eae0a1dfaca0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:48:17.650879 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.650859 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000c855f-9388-4163-9d9a-eae0a1dfaca0-kube-api-access-k54mn" (OuterVolumeSpecName: "kube-api-access-k54mn") pod "000c855f-9388-4163-9d9a-eae0a1dfaca0" (UID: "000c855f-9388-4163-9d9a-eae0a1dfaca0"). InnerVolumeSpecName "kube-api-access-k54mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:48:17.750033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.750005 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k54mn\" (UniqueName: \"kubernetes.io/projected/000c855f-9388-4163-9d9a-eae0a1dfaca0-kube-api-access-k54mn\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:48:17.750033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.750030 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-dshm\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:48:17.750227 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.750042 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/000c855f-9388-4163-9d9a-eae0a1dfaca0-tls-certs\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:48:17.750227 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.750052 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/000c855f-9388-4163-9d9a-eae0a1dfaca0-model-cache\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:48:17.971514 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.971483 2552 generic.go:356] "Generic (PLEG): container finished" podID="000c855f-9388-4163-9d9a-eae0a1dfaca0" containerID="25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a" exitCode=0 Jul 24 17:48:17.971703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.971521 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" Jul 24 17:48:17.971703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.971577 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" event={"ID":"000c855f-9388-4163-9d9a-eae0a1dfaca0","Type":"ContainerDied","Data":"25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a"} Jul 24 17:48:17.971703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.971612 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz" event={"ID":"000c855f-9388-4163-9d9a-eae0a1dfaca0","Type":"ContainerDied","Data":"c9b50d4f83bb68d0499601aa6cba92a18926d9ae39fe7cdede1d8bc960f4e41a"} Jul 24 17:48:17.971703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.971629 2552 scope.go:117] "RemoveContainer" containerID="25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a" Jul 24 17:48:17.980874 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.980859 2552 scope.go:117] "RemoveContainer" containerID="25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a" Jul 24 17:48:17.981132 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:48:17.981113 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a\": container with ID starting with 25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a not found: ID does not exist" containerID="25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a" Jul 24 17:48:17.981183 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.981142 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a"} err="failed to get container status \"25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a\": rpc error: code = NotFound desc = could not find container \"25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a\": container with ID starting with 25f1d818d49b5f8d7c5ccdf62edc2464bf4650a23202bfeb83802b44cc76979a not found: ID does not exist" Jul 24 17:48:17.992904 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.992881 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz"] Jul 24 17:48:17.996248 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:17.996226 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-028d4b/rate-test-model-028d4b-kserve-5596996455-79kqz"] Jul 24 17:48:18.964979 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:18.964942 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000c855f-9388-4163-9d9a-eae0a1dfaca0" path="/var/lib/kubelet/pods/000c855f-9388-4163-9d9a-eae0a1dfaca0/volumes" Jul 24 17:48:22.673579 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:22.673542 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:23.115453 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:23.115328 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:23.411693 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:23.411652 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:25.947706 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:25.947674 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:26.047630 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.047596 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:26.127381 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.127338 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw"] Jul 24 17:48:26.128061 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.128033 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000c855f-9388-4163-9d9a-eae0a1dfaca0" containerName="main" Jul 24 17:48:26.128061 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.128055 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="000c855f-9388-4163-9d9a-eae0a1dfaca0" containerName="main" Jul 24 17:48:26.128244 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.128218 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="000c855f-9388-4163-9d9a-eae0a1dfaca0" containerName="main" Jul 24 17:48:26.132729 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.132711 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.135032 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.135009 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert-e2e-ipp-a-052d9d\"" Jul 24 17:48:26.142997 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.142965 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw"] Jul 24 17:48:26.164677 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.164650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:26.227798 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.227708 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8d7d85-f053-4490-a77a-c135dde5a5f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.227798 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.227754 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.227994 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.227881 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjbh\" (UniqueName: \"kubernetes.io/projected/8f8d7d85-f053-4490-a77a-c135dde5a5f9-kube-api-access-7mjbh\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.329054 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.329017 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjbh\" (UniqueName: \"kubernetes.io/projected/8f8d7d85-f053-4490-a77a-c135dde5a5f9-kube-api-access-7mjbh\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.329272 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.329129 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8d7d85-f053-4490-a77a-c135dde5a5f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.329272 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.329160 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.329405 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:48:26.329314 2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert-e2e-ipp-a-052d9d: secret "maas-api-serving-cert-e2e-ipp-a-052d9d" not found Jul 24 17:48:26.329405 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:48:26.329394 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls podName:8f8d7d85-f053-4490-a77a-c135dde5a5f9 nodeName:}" failed. No retries permitted until 2026-07-24 17:48:26.82937326 +0000 UTC m=+2724.327775802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls") pod "maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" (UID: "8f8d7d85-f053-4490-a77a-c135dde5a5f9") : secret "maas-api-serving-cert-e2e-ipp-a-052d9d" not found Jul 24 17:48:26.329992 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.329965 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8d7d85-f053-4490-a77a-c135dde5a5f9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.340950 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.340928 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjbh\" (UniqueName: \"kubernetes.io/projected/8f8d7d85-f053-4490-a77a-c135dde5a5f9-kube-api-access-7mjbh\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.572501 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.572396 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:26.723552 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.723508 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:26.809829 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.809796 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:26.835441 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.835344 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:26.837818 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:26.837789 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:27.046160 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:27.046129 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:27.189028 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:27.189003 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw"] Jul 24 17:48:27.191373 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:48:27.191345 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8d7d85_f053_4490_a77a_c135dde5a5f9.slice/crio-72f052327d53fdeee6d2f556883de11277d766b0a3a8cd946593d2879e5bde68 WatchSource:0}: Error finding container 72f052327d53fdeee6d2f556883de11277d766b0a3a8cd946593d2879e5bde68: Status 404 returned error can't find the container with id 72f052327d53fdeee6d2f556883de11277d766b0a3a8cd946593d2879e5bde68 Jul 24 17:48:27.193204 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:27.193185 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:48:28.024966 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:28.024872 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" event={"ID":"8f8d7d85-f053-4490-a77a-c135dde5a5f9","Type":"ContainerStarted","Data":"c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047"} Jul 24 17:48:28.024966 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:28.024929 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" event={"ID":"8f8d7d85-f053-4490-a77a-c135dde5a5f9","Type":"ContainerStarted","Data":"72f052327d53fdeee6d2f556883de11277d766b0a3a8cd946593d2879e5bde68"} Jul 24 17:48:28.025187 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:28.025035 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:28.042907 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:28.042848 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" podStartSLOduration=1.56008627 podStartE2EDuration="2.04283117s" totalImagesPullingTime="482.7449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:28.041168944 +0000 UTC m=+2725.539571489" watchObservedRunningTime="2026-07-24 17:48:28.04283117 +0000 UTC m=+2725.541233715" Jul 24 17:48:30.490773 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:30.490733 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:34.036228 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:34.036195 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:48:41.410960 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.410926 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:41.708057 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.707972 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl"] Jul 24 17:48:41.711663 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.711645 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.715084 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.715062 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b77c4b-openshift-default-dockercfg-45c2h\"" Jul 24 17:48:41.729126 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.729105 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl"] Jul 24 17:48:41.744184 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.744158 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:41.770056 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770031 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-envoy\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770165 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770071 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-data\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770165 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770096 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-socket\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770165 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770126 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8cr\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-kube-api-access-hh8cr\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770300 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770185 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e273a682-6dec-4db1-b869-9645f873d2e0-istio-podinfo\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770300 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770210 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-istio-token\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770300 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770232 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-credential-socket\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770424 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770314 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e273a682-6dec-4db1-b869-9645f873d2e0-istiod-ca-cert\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.770424 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.770365 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-certs\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871123 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871086 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-istio-token\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871123 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871129 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-credential-socket\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871153 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e273a682-6dec-4db1-b869-9645f873d2e0-istiod-ca-cert\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871182 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-certs\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871207 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-envoy\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871229 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-data\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871255 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-socket\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871370 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871285 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8cr\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-kube-api-access-hh8cr\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871730 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871373 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e273a682-6dec-4db1-b869-9645f873d2e0-istio-podinfo\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871730 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871600 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-credential-socket\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871730 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871622 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-certs\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871863 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871738 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-data\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.871863 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.871838 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-socket\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.872139 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.872118 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e273a682-6dec-4db1-b869-9645f873d2e0-istiod-ca-cert\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.873522 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.873502 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-envoy\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.873868 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.873852 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e273a682-6dec-4db1-b869-9645f873d2e0-istio-podinfo\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.879977 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.879954 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8cr\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-kube-api-access-hh8cr\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.881790 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.881772 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-istio-token\") pod \"e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:41.926212 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:41.926185 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:42.023683 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:42.023595 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:42.178864 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:42.175346 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl"] Jul 24 17:48:42.181808 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:42.181769 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:48:42.181913 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:42.181844 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:48:42.181913 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:42.181884 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:48:43.103768 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:43.103733 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" event={"ID":"e273a682-6dec-4db1-b869-9645f873d2e0","Type":"ContainerStarted","Data":"02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d"} Jul 24 17:48:43.104152 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:43.103784 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" event={"ID":"e273a682-6dec-4db1-b869-9645f873d2e0","Type":"ContainerStarted","Data":"b10824d2d33ced11744de8b4bc21a02cae8ee3f573a5391cdc987d5b34221f44"} Jul 24 17:48:43.126571 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:43.126520 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" podStartSLOduration=2.126503581 podStartE2EDuration="2.126503581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:43.123206756 +0000 UTC m=+2740.621609303" watchObservedRunningTime="2026-07-24 17:48:43.126503581 +0000 UTC m=+2740.624906127" Jul 24 17:48:44.024594 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:44.024557 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:44.029930 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:44.029899 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:44.108766 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:44.108738 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:44.109610 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:44.109589 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:48:45.085805 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:45.085772 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:45.175101 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:45.175068 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:45.332979 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:45.332937 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:45.432659 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:45.432628 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:46.005742 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:46.005707 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:46.104056 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:46.104018 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:49.612698 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:49.612663 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:59.110296 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.110260 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:48:59.110747 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.110312 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 24 17:48:59.140002 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.139971 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:48:59.188060 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.188028 2552 generic.go:356] "Generic (PLEG): container finished" podID="e273a682-6dec-4db1-b869-9645f873d2e0" containerID="02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d" exitCode=0 Jul 24 17:48:59.188223 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.188071 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" event={"ID":"e273a682-6dec-4db1-b869-9645f873d2e0","Type":"ContainerDied","Data":"02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d"} Jul 24 17:48:59.188496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.188480 2552 scope.go:117] "RemoveContainer" containerID="02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d" Jul 24 17:48:59.189626 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.189598 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:48:59.189691 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.189667 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:48:59.189729 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:48:59.189696 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 17:49:00.196000 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:00.195968 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" event={"ID":"e273a682-6dec-4db1-b869-9645f873d2e0","Type":"ContainerStarted","Data":"0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d"} Jul 24 17:49:01.024326 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:01.024296 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:49:01.024799 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:01.024769 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:49:01.025005 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:01.024981 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 24 17:49:02.024430 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:02.024366 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:49:02.025031 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:02.024998 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 24 17:49:03.024230 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:03.024186 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 24 17:49:03.024455 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:03.024250 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 24 17:49:04.029664 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:04.029628 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:49:04.030061 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:04.029982 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:49:04.030636 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:04.030613 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:49:06.799502 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:06.799471 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-95zvq_c659f64a-f380-4809-be1e-1f1f77931d8c/payload-processing/0.log" Jul 24 17:49:19.425436 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:19.425385 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:49:26.221591 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:49:26.221512 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:04.142633 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:04.142603 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-95zvq_c659f64a-f380-4809-be1e-1f1f77931d8c/payload-processing/0.log" Jul 24 17:50:04.546024 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:04.545994 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:04.868436 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:04.868274 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:05.188557 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:05.188527 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:07.476231 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:07.476198 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:07.581255 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:07.581219 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:07.664932 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:07.664897 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:08.079719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:08.079687 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:08.251078 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:08.251047 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:08.350736 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:08.350659 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:12.007063 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:12.007030 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:33.420847 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:33.420808 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:33.520555 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:33.520517 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:38.052426 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:38.052370 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:38.430730 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:38.430701 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:44.585227 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:44.585192 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:44.738838 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:44.738809 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:44.922248 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:44.922212 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:50:45.127879 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:50:45.127842 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:04.052712 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.052621 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw"] Jul 24 17:51:04.053343 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.052927 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" podUID="8f8d7d85-f053-4490-a77a-c135dde5a5f9" containerName="maas-api" containerID="cri-o://c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047" gracePeriod=30 Jul 24 17:51:04.094863 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.094833 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:04.191862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.191813 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:04.261226 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.261194 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:04.328887 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.328348 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:51:04.429828 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.429792 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8d7d85-f053-4490-a77a-c135dde5a5f9-openshift-service-ca\") pod \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " Jul 24 17:51:04.430020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.429887 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls\") pod \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " Jul 24 17:51:04.430020 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.429970 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mjbh\" (UniqueName: \"kubernetes.io/projected/8f8d7d85-f053-4490-a77a-c135dde5a5f9-kube-api-access-7mjbh\") pod \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\" (UID: \"8f8d7d85-f053-4490-a77a-c135dde5a5f9\") " Jul 24 17:51:04.430198 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.430167 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8d7d85-f053-4490-a77a-c135dde5a5f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8f8d7d85-f053-4490-a77a-c135dde5a5f9" (UID: "8f8d7d85-f053-4490-a77a-c135dde5a5f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:51:04.430341 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.430319 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f8d7d85-f053-4490-a77a-c135dde5a5f9-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:51:04.432578 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.432547 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8f8d7d85-f053-4490-a77a-c135dde5a5f9" (UID: "8f8d7d85-f053-4490-a77a-c135dde5a5f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:51:04.432578 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.432569 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8d7d85-f053-4490-a77a-c135dde5a5f9-kube-api-access-7mjbh" (OuterVolumeSpecName: "kube-api-access-7mjbh") pod "8f8d7d85-f053-4490-a77a-c135dde5a5f9" (UID: "8f8d7d85-f053-4490-a77a-c135dde5a5f9"). InnerVolumeSpecName "kube-api-access-7mjbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:51:04.531671 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.531616 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f8d7d85-f053-4490-a77a-c135dde5a5f9-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:51:04.531671 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.531672 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjbh\" (UniqueName: \"kubernetes.io/projected/8f8d7d85-f053-4490-a77a-c135dde5a5f9-kube-api-access-7mjbh\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:51:04.859764 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.859728 2552 generic.go:356] "Generic (PLEG): container finished" podID="8f8d7d85-f053-4490-a77a-c135dde5a5f9" containerID="c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047" exitCode=0 Jul 24 17:51:04.859950 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.859774 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" Jul 24 17:51:04.859950 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.859814 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" event={"ID":"8f8d7d85-f053-4490-a77a-c135dde5a5f9","Type":"ContainerDied","Data":"c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047"} Jul 24 17:51:04.859950 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.859864 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw" event={"ID":"8f8d7d85-f053-4490-a77a-c135dde5a5f9","Type":"ContainerDied","Data":"72f052327d53fdeee6d2f556883de11277d766b0a3a8cd946593d2879e5bde68"} Jul 24 17:51:04.859950 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.859887 2552 scope.go:117] "RemoveContainer" containerID="c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047" Jul 24 17:51:04.869001 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.868986 2552 scope.go:117] "RemoveContainer" containerID="c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047" Jul 24 17:51:04.869269 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:51:04.869249 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047\": container with ID starting with c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047 not found: ID does not exist" containerID="c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047" Jul 24 17:51:04.869319 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.869277 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047"} err="failed to get container status \"c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047\": rpc error: code = NotFound desc = could not find container \"c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047\": container with ID starting with c347e1f261a893eeaefa58372126b8bd2cb37f3f724135cc143334b67ecba047 not found: ID does not exist" Jul 24 17:51:04.880846 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.880823 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw"] Jul 24 17:51:04.890381 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.890361 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-052d9d-f9f7c77f8-7z4mw"] Jul 24 17:51:04.969693 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:04.969662 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f8d7d85-f053-4490-a77a-c135dde5a5f9" path="/var/lib/kubelet/pods/8f8d7d85-f053-4490-a77a-c135dde5a5f9/volumes" Jul 24 17:51:08.657652 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:08.657618 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:09.175936 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:09.175902 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:14.169054 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:14.169008 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:43.786464 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:43.786402 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:50.963677 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:50.963630 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:51.188179 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:51.188128 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:51.287992 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:51.287914 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:51.519463 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:51.519402 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:51:57.246585 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:51:57.246538 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:03.280920 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:03.280874 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:03.360007 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:03.359975 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:03.572871 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:03.572783 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:03.789684 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:03.789647 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:14.351533 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:14.351500 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:14.427070 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:14.427032 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:16.463496 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:16.463462 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:16.549983 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:16.549947 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:19.008846 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:19.008817 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:19.385741 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:19.385648 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:25.394666 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:25.394591 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:25.498038 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:25.498008 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:25.638885 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:25.638854 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl"] Jul 24 17:52:25.639067 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:25.639052 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" containerID="cri-o://0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d" gracePeriod=30 Jul 24 17:52:25.663821 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:25.663794 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:25.898615 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:25.898556 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:30.892253 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:30.892231 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:52:31.010742 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010708 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e273a682-6dec-4db1-b869-9645f873d2e0-istio-podinfo\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.010742 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010752 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e273a682-6dec-4db1-b869-9645f873d2e0-istiod-ca-cert\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010793 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-certs\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010832 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-socket\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010856 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-data\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010884 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-credential-socket\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010947 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh8cr\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-kube-api-access-hh8cr\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011004 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.010976 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-envoy\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011318 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011016 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-istio-token\") pod \"e273a682-6dec-4db1-b869-9645f873d2e0\" (UID: \"e273a682-6dec-4db1-b869-9645f873d2e0\") " Jul 24 17:52:31.011318 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011133 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:31.011318 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011209 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e273a682-6dec-4db1-b869-9645f873d2e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:31.011318 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011163 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:31.011572 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011391 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e273a682-6dec-4db1-b869-9645f873d2e0-istiod-ca-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.011572 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011431 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-certs\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.011572 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011546 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:31.011572 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011447 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-workload-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.011874 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.011848 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:31.013206 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.013183 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:52:31.013206 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.013193 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:31.013435 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.013396 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e273a682-6dec-4db1-b869-9645f873d2e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:52:31.013502 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.013435 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-kube-api-access-hh8cr" (OuterVolumeSpecName: "kube-api-access-hh8cr") pod "e273a682-6dec-4db1-b869-9645f873d2e0" (UID: "e273a682-6dec-4db1-b869-9645f873d2e0"). InnerVolumeSpecName "kube-api-access-hh8cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:31.112830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.112798 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-istio-token\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.112830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.112825 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e273a682-6dec-4db1-b869-9645f873d2e0-istio-podinfo\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.112830 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.112835 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-data\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.113054 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.112845 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-credential-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.113054 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.112854 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh8cr\" (UniqueName: \"kubernetes.io/projected/e273a682-6dec-4db1-b869-9645f873d2e0-kube-api-access-hh8cr\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.113054 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.112862 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e273a682-6dec-4db1-b869-9645f873d2e0-istio-envoy\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:31.305491 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.305379 2552 generic.go:356] "Generic (PLEG): container finished" podID="e273a682-6dec-4db1-b869-9645f873d2e0" containerID="0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d" exitCode=0 Jul 24 17:52:31.305649 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.305490 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" Jul 24 17:52:31.305649 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.305510 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" event={"ID":"e273a682-6dec-4db1-b869-9645f873d2e0","Type":"ContainerDied","Data":"0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d"} Jul 24 17:52:31.305649 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.305572 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl" event={"ID":"e273a682-6dec-4db1-b869-9645f873d2e0","Type":"ContainerDied","Data":"b10824d2d33ced11744de8b4bc21a02cae8ee3f573a5391cdc987d5b34221f44"} Jul 24 17:52:31.305649 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.305603 2552 scope.go:117] "RemoveContainer" containerID="0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d" Jul 24 17:52:31.315697 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.315677 2552 scope.go:117] "RemoveContainer" containerID="02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d" Jul 24 17:52:31.324128 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.324109 2552 scope.go:117] "RemoveContainer" containerID="0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d" Jul 24 17:52:31.324371 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:52:31.324349 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d\": container with ID starting with 0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d not found: ID does not exist" containerID="0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d" Jul 24 17:52:31.324447 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.324380 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d"} err="failed to get container status \"0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d\": rpc error: code = NotFound desc = could not find container \"0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d\": container with ID starting with 0d52827193072ecadf8aec4053c3dd4823f81692654df428c0185ac30315394d not found: ID does not exist" Jul 24 17:52:31.324447 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.324398 2552 scope.go:117] "RemoveContainer" containerID="02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d" Jul 24 17:52:31.324686 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:52:31.324668 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d\": container with ID starting with 02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d not found: ID does not exist" containerID="02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d" Jul 24 17:52:31.324734 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.324691 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d"} err="failed to get container status \"02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d\": rpc error: code = NotFound desc = could not find container \"02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d\": container with ID starting with 02150d32a53a1fffc253de2425ec757afd8643bfba94d6d3a66117d2a3a6085d not found: ID does not exist" Jul 24 17:52:31.331365 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.331345 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl"] Jul 24 17:52:31.341608 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:31.337917 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b77c4b-openshift-default-7959ffccd-hs7jl"] Jul 24 17:52:32.965315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:32.965286 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" path="/var/lib/kubelet/pods/e273a682-6dec-4db1-b869-9645f873d2e0/volumes" Jul 24 17:52:36.557084 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.557054 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5"] Jul 24 17:52:36.557581 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.557320 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" podUID="268af474-9c86-4f56-9e19-8e6fef98b8c3" containerName="maas-api" containerID="cri-o://465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9" gracePeriod=30 Jul 24 17:52:36.595628 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.595584 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:36.674035 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.674011 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:36.775356 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.775306 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:36.821077 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.820999 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:52:36.970635 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.970603 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sbrq\" (UniqueName: \"kubernetes.io/projected/268af474-9c86-4f56-9e19-8e6fef98b8c3-kube-api-access-6sbrq\") pod \"268af474-9c86-4f56-9e19-8e6fef98b8c3\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " Jul 24 17:52:36.970816 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.970641 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls\") pod \"268af474-9c86-4f56-9e19-8e6fef98b8c3\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " Jul 24 17:52:36.970816 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.970691 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268af474-9c86-4f56-9e19-8e6fef98b8c3-openshift-service-ca\") pod \"268af474-9c86-4f56-9e19-8e6fef98b8c3\" (UID: \"268af474-9c86-4f56-9e19-8e6fef98b8c3\") " Jul 24 17:52:36.971098 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.971073 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/268af474-9c86-4f56-9e19-8e6fef98b8c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "268af474-9c86-4f56-9e19-8e6fef98b8c3" (UID: "268af474-9c86-4f56-9e19-8e6fef98b8c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:36.972719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.972689 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/268af474-9c86-4f56-9e19-8e6fef98b8c3-kube-api-access-6sbrq" (OuterVolumeSpecName: "kube-api-access-6sbrq") pod "268af474-9c86-4f56-9e19-8e6fef98b8c3" (UID: "268af474-9c86-4f56-9e19-8e6fef98b8c3"). InnerVolumeSpecName "kube-api-access-6sbrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:36.972828 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:36.972785 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "268af474-9c86-4f56-9e19-8e6fef98b8c3" (UID: "268af474-9c86-4f56-9e19-8e6fef98b8c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:52:37.072189 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.072110 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sbrq\" (UniqueName: \"kubernetes.io/projected/268af474-9c86-4f56-9e19-8e6fef98b8c3-kube-api-access-6sbrq\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:37.072189 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.072136 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/268af474-9c86-4f56-9e19-8e6fef98b8c3-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:37.072189 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.072150 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/268af474-9c86-4f56-9e19-8e6fef98b8c3-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:37.338652 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.338555 2552 generic.go:356] "Generic (PLEG): container finished" podID="268af474-9c86-4f56-9e19-8e6fef98b8c3" containerID="465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9" exitCode=0 Jul 24 17:52:37.338652 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.338623 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" Jul 24 17:52:37.338652 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.338640 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" event={"ID":"268af474-9c86-4f56-9e19-8e6fef98b8c3","Type":"ContainerDied","Data":"465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9"} Jul 24 17:52:37.338930 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.338728 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5" event={"ID":"268af474-9c86-4f56-9e19-8e6fef98b8c3","Type":"ContainerDied","Data":"69dde35aa00affb4650f093c2d59316e235f752d2e0206396b75b0e062df1ba9"} Jul 24 17:52:37.338930 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.338754 2552 scope.go:117] "RemoveContainer" containerID="465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9" Jul 24 17:52:37.348182 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.348166 2552 scope.go:117] "RemoveContainer" containerID="465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9" Jul 24 17:52:37.348454 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:52:37.348437 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9\": container with ID starting with 465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9 not found: ID does not exist" containerID="465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9" Jul 24 17:52:37.348497 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.348462 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9"} err="failed to get container status \"465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9\": rpc error: code = NotFound desc = could not find container \"465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9\": container with ID starting with 465024b5c5b1e16ef9f08c9b865854aa39664232f452b984b8a1bb2d2f34aec9 not found: ID does not exist" Jul 24 17:52:37.360490 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.360467 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5"] Jul 24 17:52:37.364089 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:37.364070 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-028d4b-c95d4b7c6-7mkm5"] Jul 24 17:52:38.964776 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:38.964744 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="268af474-9c86-4f56-9e19-8e6fef98b8c3" path="/var/lib/kubelet/pods/268af474-9c86-4f56-9e19-8e6fef98b8c3/volumes" Jul 24 17:52:41.123232 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:41.123196 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:41.490650 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:41.490609 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:48.012734 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:48.012702 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:48.078485 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:48.078452 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:48.305708 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:48.305619 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:48.514033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:48.514001 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:59.187049 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.187015 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz"] Jul 24 17:52:59.187591 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.187273 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" podUID="fe159605-888f-48d6-a81b-0959cf4adf4f" containerName="maas-api" containerID="cri-o://4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0" gracePeriod=30 Jul 24 17:52:59.225380 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.225350 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:59.294723 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.294654 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:59.395504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.395469 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:52:59.447143 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.447068 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:52:59.459600 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.459570 2552 generic.go:356] "Generic (PLEG): container finished" podID="fe159605-888f-48d6-a81b-0959cf4adf4f" containerID="4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0" exitCode=0 Jul 24 17:52:59.459754 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.459615 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" Jul 24 17:52:59.459754 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.459661 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" event={"ID":"fe159605-888f-48d6-a81b-0959cf4adf4f","Type":"ContainerDied","Data":"4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0"} Jul 24 17:52:59.459754 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.459725 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz" event={"ID":"fe159605-888f-48d6-a81b-0959cf4adf4f","Type":"ContainerDied","Data":"725b4eb6c47b8b58b62fbf92c928357c2f13a6ee076f4fa27f5aab063974eea1"} Jul 24 17:52:59.459878 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.459756 2552 scope.go:117] "RemoveContainer" containerID="4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0" Jul 24 17:52:59.470069 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.470051 2552 scope.go:117] "RemoveContainer" containerID="4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0" Jul 24 17:52:59.470350 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:52:59.470324 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0\": container with ID starting with 4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0 not found: ID does not exist" containerID="4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0" Jul 24 17:52:59.470432 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.470362 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0"} err="failed to get container status \"4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0\": rpc error: code = NotFound desc = could not find container \"4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0\": container with ID starting with 4a00b7fd142ffea4197ff36407d0d7edcaaa231303d982f432ae8b50aca507f0 not found: ID does not exist" Jul 24 17:52:59.480801 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.480776 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2r6z\" (UniqueName: \"kubernetes.io/projected/fe159605-888f-48d6-a81b-0959cf4adf4f-kube-api-access-p2r6z\") pod \"fe159605-888f-48d6-a81b-0959cf4adf4f\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " Jul 24 17:52:59.480934 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.480918 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe159605-888f-48d6-a81b-0959cf4adf4f-openshift-service-ca\") pod \"fe159605-888f-48d6-a81b-0959cf4adf4f\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " Jul 24 17:52:59.480988 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.480956 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe159605-888f-48d6-a81b-0959cf4adf4f-maas-api-tls\") pod \"fe159605-888f-48d6-a81b-0959cf4adf4f\" (UID: \"fe159605-888f-48d6-a81b-0959cf4adf4f\") " Jul 24 17:52:59.481315 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.481284 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe159605-888f-48d6-a81b-0959cf4adf4f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe159605-888f-48d6-a81b-0959cf4adf4f" (UID: "fe159605-888f-48d6-a81b-0959cf4adf4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:52:59.482891 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.482869 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe159605-888f-48d6-a81b-0959cf4adf4f-kube-api-access-p2r6z" (OuterVolumeSpecName: "kube-api-access-p2r6z") pod "fe159605-888f-48d6-a81b-0959cf4adf4f" (UID: "fe159605-888f-48d6-a81b-0959cf4adf4f"). InnerVolumeSpecName "kube-api-access-p2r6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:52:59.483007 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.482985 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe159605-888f-48d6-a81b-0959cf4adf4f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe159605-888f-48d6-a81b-0959cf4adf4f" (UID: "fe159605-888f-48d6-a81b-0959cf4adf4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:52:59.581719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.581679 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2r6z\" (UniqueName: \"kubernetes.io/projected/fe159605-888f-48d6-a81b-0959cf4adf4f-kube-api-access-p2r6z\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:59.581719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.581712 2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe159605-888f-48d6-a81b-0959cf4adf4f-openshift-service-ca\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:59.581719 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.581727 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe159605-888f-48d6-a81b-0959cf4adf4f-maas-api-tls\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:52:59.782880 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.782849 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz"] Jul 24 17:52:59.786866 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:52:59.786844 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a11aa7-547b8ddc9c-2bzwz"] Jul 24 17:53:00.964735 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:00.964702 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe159605-888f-48d6-a81b-0959cf4adf4f" path="/var/lib/kubelet/pods/fe159605-888f-48d6-a81b-0959cf4adf4f/volumes" Jul 24 17:53:03.173138 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:03.173111 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:53:03.183296 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:03.183273 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:53:03.715458 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:03.715401 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:04.103176 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:04.103082 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:10.273655 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:10.273619 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:10.373593 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:10.373558 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:10.526330 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:10.526246 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4"] Jul 24 17:53:10.526940 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:10.526622 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" containerID="cri-o://406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667" gracePeriod=30 Jul 24 17:53:10.561751 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:10.561716 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:10.784538 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:10.784453 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:11.184171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:11.184138 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59b78ff696-gd8c7_62bce08a-c14d-469c-9605-c9aee71fcf3b/authorino/0.log" Jul 24 17:53:15.785967 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.785939 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:53:15.934922 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.934886 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-credential-socket\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.934959 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-istio-token\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.934993 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24414c34-102b-416b-a4c5-0df50cad0a52-istio-podinfo\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935037 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-data\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935104 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935079 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzdmb\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-kube-api-access-jzdmb\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935108 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-socket\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935131 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:53:15.935316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935168 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-envoy\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935208 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24414c34-102b-416b-a4c5-0df50cad0a52-istiod-ca-cert\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935235 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-certs\") pod \"24414c34-102b-416b-a4c5-0df50cad0a52\" (UID: \"24414c34-102b-416b-a4c5-0df50cad0a52\") " Jul 24 17:53:15.935617 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935382 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:53:15.935617 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935588 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:15.935617 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935609 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-credential-socket\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:15.935772 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935696 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24414c34-102b-416b-a4c5-0df50cad0a52-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:53:15.935838 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935794 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:53:15.935949 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.935917 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-data" (OuterVolumeSpecName: "istio-data") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:53:15.937243 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.937221 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-istio-token" (OuterVolumeSpecName: "istio-token") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:53:15.937474 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.937455 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:53:15.937650 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.937628 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24414c34-102b-416b-a4c5-0df50cad0a52-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 17:53:15.937745 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:15.937726 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-kube-api-access-jzdmb" (OuterVolumeSpecName: "kube-api-access-jzdmb") pod "24414c34-102b-416b-a4c5-0df50cad0a52" (UID: "24414c34-102b-416b-a4c5-0df50cad0a52"). InnerVolumeSpecName "kube-api-access-jzdmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:53:16.037033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.037001 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdmb\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-kube-api-access-jzdmb\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:16.037033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.037026 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-envoy\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:16.037033 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.037037 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24414c34-102b-416b-a4c5-0df50cad0a52-istiod-ca-cert\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:16.037251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.037045 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-workload-certs\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:16.037251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.037053 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24414c34-102b-416b-a4c5-0df50cad0a52-istio-token\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:16.037251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.037062 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24414c34-102b-416b-a4c5-0df50cad0a52-istio-podinfo\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:16.037251 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.037070 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24414c34-102b-416b-a4c5-0df50cad0a52-istio-data\") on node \"ip-10-0-138-185.ec2.internal\" DevicePath \"\"" Jul 24 17:53:16.557429 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.557376 2552 generic.go:356] "Generic (PLEG): container finished" podID="24414c34-102b-416b-a4c5-0df50cad0a52" containerID="406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667" exitCode=0 Jul 24 17:53:16.557625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.557457 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" Jul 24 17:53:16.557625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.557453 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" event={"ID":"24414c34-102b-416b-a4c5-0df50cad0a52","Type":"ContainerDied","Data":"406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667"} Jul 24 17:53:16.557625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.557588 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4" event={"ID":"24414c34-102b-416b-a4c5-0df50cad0a52","Type":"ContainerDied","Data":"8c44e08daf90c97daba88eb09df263e0150610c771486d2a7e44d22749ad8d7c"} Jul 24 17:53:16.557625 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.557620 2552 scope.go:117] "RemoveContainer" containerID="406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667" Jul 24 17:53:16.566903 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.566888 2552 scope.go:117] "RemoveContainer" containerID="1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786" Jul 24 17:53:16.574853 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.574836 2552 scope.go:117] "RemoveContainer" containerID="406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667" Jul 24 17:53:16.575127 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:53:16.575105 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667\": container with ID starting with 406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667 not found: ID does not exist" containerID="406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667" Jul 24 17:53:16.575200 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.575139 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667"} err="failed to get container status \"406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667\": rpc error: code = NotFound desc = could not find container \"406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667\": container with ID starting with 406ebf5c1a96ca7bc5120a5bae21cbf0ff070e4b1752dab56174f3738599d667 not found: ID does not exist" Jul 24 17:53:16.575200 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.575164 2552 scope.go:117] "RemoveContainer" containerID="1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786" Jul 24 17:53:16.575438 ip-10-0-138-185 kubenswrapper[2552]: E0724 17:53:16.575401 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786\": container with ID starting with 1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786 not found: ID does not exist" containerID="1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786" Jul 24 17:53:16.575438 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.575441 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786"} err="failed to get container status \"1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786\": rpc error: code = NotFound desc = could not find container \"1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786\": container with ID starting with 1239d80d521023ddd3a697930a51802104013d6dec6f7e97eec9a1820128f786 not found: ID does not exist" Jul 24 17:53:16.578279 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.578256 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4"] Jul 24 17:53:16.583448 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.583427 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-a11aa7-openshift-default-855c9d8cc6-sv4p4"] Jul 24 17:53:16.630378 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.630352 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rtpmx_202119ad-4258-4448-9210-a436ba05064a/manager/0.log" Jul 24 17:53:16.890691 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.890614 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9wwg9_cba587b9-b59e-4278-9df4-bc874d9414a1/server/0.log" Jul 24 17:53:16.964870 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:16.964831 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" path="/var/lib/kubelet/pods/24414c34-102b-416b-a4c5-0df50cad0a52/volumes" Jul 24 17:53:17.004212 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:17.004184 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rs69p_39d73d61-c5d6-49e6-99e4-679136058f81/manager/0.log" Jul 24 17:53:17.121368 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:17.121344 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bc8c4649b-6dlvf_81f2ef33-dd41-4967-82d7-ccc90d86c242/copy-manifests/0.log" Jul 24 17:53:17.128612 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:17.128594 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bc8c4649b-6dlvf_81f2ef33-dd41-4967-82d7-ccc90d86c242/manager/0.log" Jul 24 17:53:17.723814 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:17.723784 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c897c7bdd-8nqkd_92269590-ad86-4c74-bfa9-9bbe70064fee/maas-api/0.log" Jul 24 17:53:19.435037 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:19.435001 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59b78ff696-gd8c7_62bce08a-c14d-469c-9605-c9aee71fcf3b/authorino/0.log" Jul 24 17:53:19.549161 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:19.549116 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7pks2_480bfbf3-8300-420a-b091-f34e5118bbfa/manager/0.log" Jul 24 17:53:19.657449 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:19.657421 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q7kss_4154d801-e622-49fb-8ec7-6d231864e233/manager/0.log" Jul 24 17:53:19.874959 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:19.874882 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dq7tx_503b0002-85f0-453f-a628-deeec289abaa/registry-server/0.log" Jul 24 17:53:19.994200 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:19.994171 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gc688_06763651-dbb9-4379-a734-830ad94ac506/manager/0.log" Jul 24 17:53:20.222577 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:20.222546 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gllt4_029eaa23-1a41-4924-ab23-74d0b999d8ff/manager/0.log" Jul 24 17:53:20.576245 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:20.576160 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v_0c40c231-971d-41de-9487-e66aa463b36b/istio-proxy/0.log" Jul 24 17:53:20.910933 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:20.910906 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d6f9cf76-xx6qw_252e838b-cd9b-4247-915e-4499396af2d5/kube-auth-proxy/0.log" Jul 24 17:53:21.030171 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:21.030138 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpt5t_78961fb3-3403-4f44-a263-0e730fdb851a/istio-proxy/1.log" Jul 24 17:53:21.260580 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:21.260498 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-95zvq_c659f64a-f380-4809-be1e-1f1f77931d8c/payload-processing/0.log" Jul 24 17:53:21.374008 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:21.373979 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8cbf98d5-bg4mh_69009bfd-dd6d-4838-9981-c3b080dbadd6/router/0.log" Jul 24 17:53:22.047525 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:22.047498 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff8pzx_873a1359-6830-4840-a971-6b100082dac1/main/0.log" Jul 24 17:53:27.778619 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:27.778582 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:27.842958 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:27.842914 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:30.628134 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:30.628101 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zmnnx_b4985b5b-1bdb-4bd8-b250-67e63f89d01f/global-pull-secret-syncer/0.log" Jul 24 17:53:30.678570 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:30.678536 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" Jul 24 17:53:30.827340 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:30.827311 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-185.ec2.internal_8ea5a8b31f868b5de8cabbf89b4c2b24/haproxy/0.log" Jul 24 17:53:35.841999 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:35.841966 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-59b78ff696-gd8c7_62bce08a-c14d-469c-9605-c9aee71fcf3b/authorino/0.log" Jul 24 17:53:35.872643 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:35.872615 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7pks2_480bfbf3-8300-420a-b091-f34e5118bbfa/manager/0.log" Jul 24 17:53:35.908494 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:35.908465 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q7kss_4154d801-e622-49fb-8ec7-6d231864e233/manager/0.log" Jul 24 17:53:35.983688 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:35.983659 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-dq7tx_503b0002-85f0-453f-a628-deeec289abaa/registry-server/0.log" Jul 24 17:53:36.131205 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:36.131120 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-gc688_06763651-dbb9-4379-a734-830ad94ac506/manager/0.log" Jul 24 17:53:36.217534 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:36.217494 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gllt4_029eaa23-1a41-4924-ab23-74d0b999d8ff/manager/0.log" Jul 24 17:53:38.328475 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:38.328378 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-vf6rz_0e0f63bd-220f-46f4-b52c-60dccf2f6e2f/cluster-monitoring-operator/0.log" Jul 24 17:53:38.773539 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:38.773511 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vw9vm_58cf2dfc-d07e-4e2e-8bcb-e053a18af71b/node-exporter/0.log" Jul 24 17:53:38.794504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:38.794480 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vw9vm_58cf2dfc-d07e-4e2e-8bcb-e053a18af71b/kube-rbac-proxy/0.log" Jul 24 17:53:38.816624 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:38.816600 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vw9vm_58cf2dfc-d07e-4e2e-8bcb-e053a18af71b/init-textfile/0.log" Jul 24 17:53:40.117504 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.117466 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn"] Jul 24 17:53:40.118034 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118015 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f8d7d85-f053-4490-a77a-c135dde5a5f9" containerName="maas-api" Jul 24 17:53:40.118119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118037 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f8d7d85-f053-4490-a77a-c135dde5a5f9" containerName="maas-api" Jul 24 17:53:40.118119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118048 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" Jul 24 17:53:40.118119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118057 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" Jul 24 17:53:40.118119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118083 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="268af474-9c86-4f56-9e19-8e6fef98b8c3" containerName="maas-api" Jul 24 17:53:40.118119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118093 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="268af474-9c86-4f56-9e19-8e6fef98b8c3" containerName="maas-api" Jul 24 17:53:40.118119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118102 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" Jul 24 17:53:40.118119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118111 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118125 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe159605-888f-48d6-a81b-0959cf4adf4f" containerName="maas-api" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118134 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe159605-888f-48d6-a81b-0959cf4adf4f" containerName="maas-api" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118145 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118153 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118269 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e273a682-6dec-4db1-b869-9645f873d2e0" containerName="istio-proxy" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118284 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118294 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f8d7d85-f053-4490-a77a-c135dde5a5f9" containerName="maas-api" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118305 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="24414c34-102b-416b-a4c5-0df50cad0a52" containerName="istio-proxy" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118314 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="268af474-9c86-4f56-9e19-8e6fef98b8c3" containerName="maas-api" Jul 24 17:53:40.118510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.118326 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe159605-888f-48d6-a81b-0959cf4adf4f" containerName="maas-api" Jul 24 17:53:40.123309 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.123288 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.125795 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.125768 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jgdcp\"/\"openshift-service-ca.crt\"" Jul 24 17:53:40.125984 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.125966 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jgdcp\"/\"kube-root-ca.crt\"" Jul 24 17:53:40.126748 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.126732 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jgdcp\"/\"default-dockercfg-6zhq5\"" Jul 24 17:53:40.131142 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.131122 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn"] Jul 24 17:53:40.141215 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.141185 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppsv\" (UniqueName: \"kubernetes.io/projected/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-kube-api-access-vppsv\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.141346 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.141242 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-sys\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.141346 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.141290 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-proc\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.141346 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.141322 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-podres\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.141552 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.141377 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-lib-modules\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.242534 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242496 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-lib-modules\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.242703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242551 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppsv\" (UniqueName: \"kubernetes.io/projected/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-kube-api-access-vppsv\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.242703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242591 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-sys\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.242703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242608 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-proc\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.242703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242626 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-podres\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.242703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242672 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-lib-modules\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.242703 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242672 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-sys\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.243003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242704 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-proc\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.243003 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.242741 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-podres\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.252186 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.252157 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppsv\" (UniqueName: \"kubernetes.io/projected/96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd-kube-api-access-vppsv\") pod \"perf-node-gather-daemonset-z4zpn\" (UID: \"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd\") " pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.434556 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.434528 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.577863 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.577834 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn"] Jul 24 17:53:40.579943 ip-10-0-138-185 kubenswrapper[2552]: W0724 17:53:40.579916 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod96406a3e_2fa2_49ca_97c7_e3b8fd0e95fd.slice/crio-fef7a920f03ad3242b81af4e8a2c73ffb3c9bc7d57af56bff241b03610a7903c WatchSource:0}: Error finding container fef7a920f03ad3242b81af4e8a2c73ffb3c9bc7d57af56bff241b03610a7903c: Status 404 returned error can't find the container with id fef7a920f03ad3242b81af4e8a2c73ffb3c9bc7d57af56bff241b03610a7903c Jul 24 17:53:40.603668 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.603649 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-zsp7b_4b95cb83-2ecb-47a0-97f7-90b8a8b0b981/networking-console-plugin/0.log" Jul 24 17:53:40.682933 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.682901 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" event={"ID":"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd","Type":"ContainerStarted","Data":"403bfd44b75a8a0d6688e38e0eeb4a88497d752cf64c80161133751562657ecb"} Jul 24 17:53:40.683076 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.682956 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" event={"ID":"96406a3e-2fa2-49ca-97c7-e3b8fd0e95fd","Type":"ContainerStarted","Data":"fef7a920f03ad3242b81af4e8a2c73ffb3c9bc7d57af56bff241b03610a7903c"} Jul 24 17:53:40.683076 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.683022 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:40.702034 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:40.701940 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" podStartSLOduration=0.701924819 podStartE2EDuration="701.924819ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:53:40.700977392 +0000 UTC m=+3038.199379939" watchObservedRunningTime="2026-07-24 17:53:40.701924819 +0000 UTC m=+3038.200327363" Jul 24 17:53:41.170651 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:41.170610 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/1.log" Jul 24 17:53:41.175912 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:41.175890 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-c5bfp_9d5333a0-2111-4134-b27c-8a6d49e53bc0/console-operator/2.log" Jul 24 17:53:41.671644 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:41.671618 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-gr78g_c1e023d0-0c62-4c0e-a105-c4c18106d90e/download-server/0.log" Jul 24 17:53:42.228119 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:42.228095 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-jf4vm_998d1e8e-6a69-47c4-a035-bbbdb44d68bc/volume-data-source-validator/0.log" Jul 24 17:53:43.325145 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:43.325115 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sq9g2_45bb5108-6076-46d7-9421-ca8ea674fe31/dns/0.log" Jul 24 17:53:43.349681 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:43.349656 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sq9g2_45bb5108-6076-46d7-9421-ca8ea674fe31/kube-rbac-proxy/0.log" Jul 24 17:53:43.423057 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:43.423025 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tgptf_c68158e9-2ffc-401e-a9cd-5eb60b63d7c3/dns-node-resolver/0.log" Jul 24 17:53:44.009400 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:44.009373 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fbnqh_d239a832-beca-4842-b38f-fee778ed3b0d/node-ca/0.log" Jul 24 17:53:44.971235 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:44.971209 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v_0c40c231-971d-41de-9487-e66aa463b36b/istio-proxy/0.log" Jul 24 17:53:45.288580 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:45.288501 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d6f9cf76-xx6qw_252e838b-cd9b-4247-915e-4499396af2d5/kube-auth-proxy/0.log" Jul 24 17:53:45.334767 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:45.334743 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpt5t_78961fb3-3403-4f44-a263-0e730fdb851a/istio-proxy/0.log" Jul 24 17:53:45.366510 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:45.366488 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpt5t_78961fb3-3403-4f44-a263-0e730fdb851a/istio-proxy/1.log" Jul 24 17:53:45.476748 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:45.476723 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-95zvq_c659f64a-f380-4809-be1e-1f1f77931d8c/payload-processing/0.log" Jul 24 17:53:45.501140 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:45.501119 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8cbf98d5-bg4mh_69009bfd-dd6d-4838-9981-c3b080dbadd6/router/0.log" Jul 24 17:53:46.335164 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:46.335139 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mkptd_5d20e785-46d3-4a00-a8a6-50a30445dbc8/serve-healthcheck-canary/0.log" Jul 24 17:53:46.697890 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:46.697864 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jgdcp/perf-node-gather-daemonset-z4zpn" Jul 24 17:53:46.803368 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:46.803341 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-455nt_3265b58a-6a78-4afa-ba20-533d76387071/insights-operator/0.log" Jul 24 17:53:46.804118 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:46.804101 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-455nt_3265b58a-6a78-4afa-ba20-533d76387071/insights-operator/1.log" Jul 24 17:53:46.824074 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:46.824050 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dg95g_86216dce-69aa-4dc1-9231-85ad8505db50/kube-rbac-proxy/0.log" Jul 24 17:53:46.843775 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:46.843756 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dg95g_86216dce-69aa-4dc1-9231-85ad8505db50/exporter/0.log" Jul 24 17:53:46.864358 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:46.864333 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dg95g_86216dce-69aa-4dc1-9231-85ad8505db50/extractor/0.log" Jul 24 17:53:47.610316 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:47.610275 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:47.663728 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:47.663684 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-lrgjl_5954c515-ebf7-48b7-aca4-6dbae06907a1/konnectivity-agent/0.log" limit=1024 Jul 24 17:53:49.514638 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:49.514611 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rtpmx_202119ad-4258-4448-9210-a436ba05064a/manager/0.log" Jul 24 17:53:49.716803 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:49.716772 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9wwg9_cba587b9-b59e-4278-9df4-bc874d9414a1/server/0.log" Jul 24 17:53:49.792862 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:49.792829 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rs69p_39d73d61-c5d6-49e6-99e4-679136058f81/manager/0.log" Jul 24 17:53:49.824183 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:49.824155 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bc8c4649b-6dlvf_81f2ef33-dd41-4967-82d7-ccc90d86c242/manager/0.log" Jul 24 17:53:49.850702 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:49.850674 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bc8c4649b-6dlvf_81f2ef33-dd41-4967-82d7-ccc90d86c242/copy-manifests/0.log" Jul 24 17:53:50.819796 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:50.819762 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v_0c40c231-971d-41de-9487-e66aa463b36b/istio-proxy/0.log" Jul 24 17:53:51.172369 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:51.172332 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d6f9cf76-xx6qw_252e838b-cd9b-4247-915e-4499396af2d5/kube-auth-proxy/0.log" Jul 24 17:53:51.201617 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:51.201589 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpt5t_78961fb3-3403-4f44-a263-0e730fdb851a/istio-proxy/0.log" Jul 24 17:53:51.228214 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:51.228188 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpt5t_78961fb3-3403-4f44-a263-0e730fdb851a/istio-proxy/1.log" Jul 24 17:53:51.278400 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:51.278368 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-95zvq_c659f64a-f380-4809-be1e-1f1f77931d8c/payload-processing/0.log" Jul 24 17:53:51.297727 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:51.297702 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8cbf98d5-bg4mh_69009bfd-dd6d-4838-9981-c3b080dbadd6/router/0.log" Jul 24 17:53:52.732704 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:52.732680 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dzv2v_0c40c231-971d-41de-9487-e66aa463b36b/istio-proxy/0.log" Jul 24 17:53:53.039658 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:53.039587 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d6f9cf76-xx6qw_252e838b-cd9b-4247-915e-4499396af2d5/kube-auth-proxy/0.log" Jul 24 17:53:53.070952 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:53.070921 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpt5t_78961fb3-3403-4f44-a263-0e730fdb851a/istio-proxy/0.log" Jul 24 17:53:53.099421 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:53.099389 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpt5t_78961fb3-3403-4f44-a263-0e730fdb851a/istio-proxy/1.log" Jul 24 17:53:53.154680 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:53.154647 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-95zvq_c659f64a-f380-4809-be1e-1f1f77931d8c/payload-processing/0.log" Jul 24 17:53:53.171369 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:53.171325 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8cbf98d5-bg4mh_69009bfd-dd6d-4838-9981-c3b080dbadd6/router/0.log" Jul 24 17:53:57.211252 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:57.211220 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gjfqb_ecd808f2-451a-4b8a-a0df-0461eef9e6b5/migrator/0.log" Jul 24 17:53:57.257235 ip-10-0-138-185 kubenswrapper[2552]: I0724 17:53:57.257204 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gjfqb_ecd808f2-451a-4b8a-a0df-0461eef9e6b5/graceful-termination/0.log"