Aug 04 15:08:50.355852 ip-10-0-138-35 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Aug 04 15:08:50.355863 ip-10-0-138-35 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Aug 04 15:08:50.355869 ip-10-0-138-35 systemd[1]: kubelet.service: Failed with result 'resources'. Aug 04 15:08:50.356082 ip-10-0-138-35 systemd[1]: Failed to start Kubernetes Kubelet. Aug 04 15:09:01.967630 ip-10-0-138-35 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Aug 04 15:09:01.967648 ip-10-0-138-35 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1607f246a0dc4f8481efae825fef6091 -- Aug 04 15:10:34.530864 ip-10-0-138-35 systemd[1]: Starting Kubernetes Kubelet... Aug 04 15:10:34.958489 ip-10-0-138-35 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:10:34.958489 ip-10-0-138-35 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Aug 04 15:10:34.958489 ip-10-0-138-35 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:10:34.958489 ip-10-0-138-35 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Aug 04 15:10:34.958489 ip-10-0-138-35 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Aug 04 15:10:34.960522 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.960437 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Aug 04 15:10:34.964796 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964781 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:10:34.964796 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964795 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964799 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964802 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964805 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964807 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964810 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964813 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964816 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964819 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964822 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964825 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964828 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964831 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964833 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964837 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964842 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964845 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964848 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964850 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:10:34.964866 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964853 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964856 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964859 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964861 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964864 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964867 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964870 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964872 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964876 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964879 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964882 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964884 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964887 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964890 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964892 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964895 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964897 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964900 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964903 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964905 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:10:34.965311 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964908 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964911 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964913 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964916 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964920 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964922 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964924 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964927 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964929 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964932 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964934 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964937 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964939 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964941 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964945 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964948 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964951 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964954 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964956 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:10:34.965960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964959 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964962 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964964 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964967 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964970 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964972 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964975 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964978 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964981 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964984 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964986 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964990 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964992 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964995 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.964997 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965000 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965002 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965005 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965007 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965009 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965012 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:10:34.966751 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965014 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965017 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965019 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965022 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965027 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965030 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965035 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965038 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965042 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965046 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965048 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965051 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965054 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965057 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965060 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965062 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965065 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965068 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.965070 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:10:34.967254 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966463 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966481 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966487 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966493 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966500 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966509 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966514 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966520 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966525 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966530 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966541 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966546 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966550 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966555 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966560 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966565 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966569 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966590 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966595 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966599 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:10:34.967736 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966627 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966633 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966639 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966644 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966649 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966654 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966659 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966671 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966676 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966681 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966686 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966711 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966714 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966721 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966826 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966832 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966837 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966840 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966843 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966846 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:10:34.968225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966850 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966854 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966857 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966860 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966863 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966865 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966868 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966871 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966874 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966876 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966879 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966882 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966886 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966889 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966892 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966894 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966899 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966902 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966905 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966908 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:10:34.968739 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966912 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966915 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966918 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966921 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966924 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966926 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966929 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966931 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966934 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966938 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966941 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966944 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966947 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966949 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966953 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966955 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966958 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966962 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966965 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966969 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:10:34.969225 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966972 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966974 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966977 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966979 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966982 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966985 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966988 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966990 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966993 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966996 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.966999 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967001 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967004 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967007 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967010 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967012 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967015 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967017 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967020 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:10:34.969753 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967407 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967412 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967416 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967419 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967422 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967425 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967428 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967430 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967433 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967436 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967439 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967441 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967444 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967446 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967449 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967452 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967454 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967458 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967461 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967464 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:10:34.970245 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967467 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967469 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967472 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967475 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967478 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967480 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967483 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967485 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967487 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967490 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967492 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967495 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967497 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967500 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967502 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967505 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967509 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967512 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967515 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:10:34.970743 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967518 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967521 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967524 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967526 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967529 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967531 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967534 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967536 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967539 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967541 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967546 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967548 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967551 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967553 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967556 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967559 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967561 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967564 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967567 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967569 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:10:34.971228 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967588 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967592 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967595 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967597 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967600 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967603 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967606 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967608 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967611 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967614 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967616 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967618 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967621 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967625 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967627 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967630 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967633 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967635 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967638 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967640 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:10:34.971735 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967643 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967645 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967649 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967653 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967656 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967659 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967661 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967664 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967667 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967670 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967672 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967675 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967677 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967680 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967682 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967685 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967689 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967691 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967694 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:10:34.972267 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.967696 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967767 2559 flags.go:64] FLAG: --address="0.0.0.0" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967774 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967781 2559 flags.go:64] FLAG: --anonymous-auth="true" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967785 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967789 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967792 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967796 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967800 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967804 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967807 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967810 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967813 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967817 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967820 2559 flags.go:64] FLAG: --cgroup-root="" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967823 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967826 2559 flags.go:64] FLAG: --client-ca-file="" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967830 2559 flags.go:64] FLAG: --cloud-provider="external" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967833 2559 flags.go:64] FLAG: --cluster-dns="[]" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967845 2559 flags.go:64] FLAG: --cluster-domain="" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967849 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967852 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967855 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967858 2559 flags.go:64] FLAG: --container-log-max-files="5" Aug 04 15:10:34.972751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967862 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967865 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967868 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967871 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967874 2559 flags.go:64] FLAG: --contention-profiling="false" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967877 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967880 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967884 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967886 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967890 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967894 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967897 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967900 2559 flags.go:64] FLAG: --enable-load-reader="false" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967903 2559 flags.go:64] FLAG: --enable-server="true" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967905 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967910 2559 flags.go:64] FLAG: --event-burst="100" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967913 2559 flags.go:64] FLAG: --event-qps="50" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967915 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967918 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967921 2559 flags.go:64] FLAG: --eviction-hard="" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967925 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967927 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967931 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967934 2559 flags.go:64] FLAG: --eviction-soft="" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967937 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Aug 04 15:10:34.973318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967940 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967944 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967948 2559 flags.go:64] FLAG: --experimental-mounter-path="" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967951 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967954 2559 flags.go:64] FLAG: --fail-swap-on="true" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967956 2559 flags.go:64] FLAG: --feature-gates="" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967965 2559 flags.go:64] FLAG: --file-check-frequency="20s" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967968 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967971 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967974 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967977 2559 flags.go:64] FLAG: --healthz-port="10248" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967980 2559 flags.go:64] FLAG: --help="false" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967983 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-138-35.ec2.internal" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967986 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967989 2559 flags.go:64] FLAG: --http-check-frequency="20s" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967992 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967995 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.967999 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968002 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968005 2559 flags.go:64] FLAG: --image-service-endpoint="" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968007 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968010 2559 flags.go:64] FLAG: --kube-api-burst="100" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968013 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968017 2559 flags.go:64] FLAG: --kube-api-qps="50" Aug 04 15:10:34.973938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968019 2559 flags.go:64] FLAG: --kube-reserved="" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968022 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968025 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968028 2559 flags.go:64] FLAG: --kubelet-cgroups="" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968031 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968034 2559 flags.go:64] FLAG: --lock-file="" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968037 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968040 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968043 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968049 2559 flags.go:64] FLAG: --log-json-split-stream="false" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968052 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968055 2559 flags.go:64] FLAG: --log-text-split-stream="false" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968058 2559 flags.go:64] FLAG: --logging-format="text" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968061 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968064 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968067 2559 flags.go:64] FLAG: --manifest-url="" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968070 2559 flags.go:64] FLAG: --manifest-url-header="" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968074 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968077 2559 flags.go:64] FLAG: --max-open-files="1000000" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968081 2559 flags.go:64] FLAG: --max-pods="110" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968084 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968087 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968089 2559 flags.go:64] FLAG: --memory-manager-policy="None" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968092 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968095 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Aug 04 15:10:34.974556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968098 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968101 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968108 2559 flags.go:64] FLAG: --node-status-max-images="50" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968111 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968114 2559 flags.go:64] FLAG: --oom-score-adj="-999" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968117 2559 flags.go:64] FLAG: --pod-cidr="" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968120 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968126 2559 flags.go:64] FLAG: --pod-manifest-path="" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968129 2559 flags.go:64] FLAG: --pod-max-pids="-1" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968132 2559 flags.go:64] FLAG: --pods-per-core="0" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968135 2559 flags.go:64] FLAG: --port="10250" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968138 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968141 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-026d158569378e242" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968144 2559 flags.go:64] FLAG: --qos-reserved="" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968147 2559 flags.go:64] FLAG: --read-only-port="10255" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968150 2559 flags.go:64] FLAG: --register-node="true" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968153 2559 flags.go:64] FLAG: --register-with-taints="" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968156 2559 flags.go:64] FLAG: --registry-burst="10" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968160 2559 flags.go:64] FLAG: --registry-qps="5" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968163 2559 flags.go:64] FLAG: --reserved-cpus="" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968166 2559 flags.go:64] FLAG: --reserved-memory="" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968169 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968172 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968175 2559 flags.go:64] FLAG: --rotate-certificates="false" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968178 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Aug 04 15:10:34.975196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968180 2559 flags.go:64] FLAG: --runonce="false" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968183 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968186 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968191 2559 flags.go:64] FLAG: --seccomp-default="false" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968194 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968197 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968200 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968203 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968206 2559 flags.go:64] FLAG: --storage-driver-password="root" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968209 2559 flags.go:64] FLAG: --storage-driver-secure="false" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968212 2559 flags.go:64] FLAG: --storage-driver-table="stats" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968215 2559 flags.go:64] FLAG: --storage-driver-user="root" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968217 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968220 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968223 2559 flags.go:64] FLAG: --system-cgroups="" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968226 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968231 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968234 2559 flags.go:64] FLAG: --tls-cert-file="" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968237 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968242 2559 flags.go:64] FLAG: --tls-min-version="" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968245 2559 flags.go:64] FLAG: --tls-private-key-file="" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968247 2559 flags.go:64] FLAG: --topology-manager-policy="none" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968250 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968253 2559 flags.go:64] FLAG: --topology-manager-scope="container" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968256 2559 flags.go:64] FLAG: --v="2" Aug 04 15:10:34.975854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968260 2559 flags.go:64] FLAG: --version="false" Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968264 2559 flags.go:64] FLAG: --vmodule="" Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968268 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.968271 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968380 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968384 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968387 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968390 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968393 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968396 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968400 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968403 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968405 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968408 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968411 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968413 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968416 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968418 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968421 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968424 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:10:34.976445 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968427 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968431 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968434 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968437 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968440 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968443 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968446 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968449 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968452 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968454 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968457 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968460 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968463 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968465 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968468 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968471 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968474 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968476 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968480 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968484 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:10:34.976973 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968487 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968489 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968493 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968495 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968498 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968501 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968503 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968506 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968508 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968511 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968514 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968516 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968519 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968521 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968524 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968527 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968529 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968534 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968536 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968539 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:10:34.977461 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968541 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968544 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968548 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968552 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968555 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968557 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968560 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968563 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968566 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968569 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968586 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968591 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968594 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968597 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968601 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968603 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968606 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968609 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968611 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:10:34.977982 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968614 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968616 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968619 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968621 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968624 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968626 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968629 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968632 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968634 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968638 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968641 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968644 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968647 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968649 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968652 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968655 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968657 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968660 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968663 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:10:34.978452 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968665 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968668 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968671 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968674 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.968677 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.969315 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.975871 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.975887 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.975906 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.975911 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975965 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975970 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975973 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975977 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975980 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:10:34.978936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975982 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975986 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975989 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975991 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975994 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.975997 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976000 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976003 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976006 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976009 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976011 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976014 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976016 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976019 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976022 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976025 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976027 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976030 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976033 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976038 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:10:34.979321 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976041 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976044 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976046 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976050 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976053 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976056 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976059 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976061 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976064 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976067 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976069 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976072 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976074 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976077 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976080 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976082 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976087 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976090 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976093 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976096 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:10:34.979861 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976099 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976101 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976104 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976107 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976110 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976113 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976115 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976118 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976121 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976123 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976126 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976129 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976133 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976135 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976139 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976143 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976147 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976150 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976152 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976155 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:10:34.980350 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976157 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976160 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976163 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976165 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976168 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976171 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976173 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976177 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976181 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976184 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976187 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976190 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976192 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976195 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976197 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976200 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976203 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976206 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976208 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:10:34.980851 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976211 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976214 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976216 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976219 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976222 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976225 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976228 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976231 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976233 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976236 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976239 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976242 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976244 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976247 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976249 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:10:34.981335 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.976254 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976362 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976366 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976369 2559 feature_gate.go:328] unrecognized feature gate: Example2 Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976372 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976375 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976378 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976381 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976385 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976389 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976392 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976394 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976397 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976400 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976403 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976406 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976409 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976411 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976414 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976417 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Aug 04 15:10:34.981844 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976419 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976422 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976424 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976427 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976430 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976432 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976436 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976438 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976440 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976443 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976445 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976448 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976451 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976453 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976457 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976459 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976462 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976464 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976467 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Aug 04 15:10:34.982320 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976469 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976472 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976474 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976477 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976480 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976482 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976485 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976489 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976492 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976495 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976498 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976501 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976504 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976507 2559 feature_gate.go:328] unrecognized feature gate: Example Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976509 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976512 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976514 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976517 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976521 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Aug 04 15:10:34.982786 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976523 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976526 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976528 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976531 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976533 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976537 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976541 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976543 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976546 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976548 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976551 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976553 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976556 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976558 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976561 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976563 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976566 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976568 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976571 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Aug 04 15:10:34.983231 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976588 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976591 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976594 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976597 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976599 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976602 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976605 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976607 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976610 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976612 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976615 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976617 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976620 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976624 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976626 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976629 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976632 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976634 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976637 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976639 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Aug 04 15:10:34.983702 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976642 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976644 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:34.976647 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.976652 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.976757 2559 server.go:962] "Client rotation is on, will bootstrap in background" Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.978811 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.980246 2559 server.go:1019] "Starting client certificate rotation" Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.980342 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.981235 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Aug 04 15:10:34.984202 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.983972 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 15:10:34.988047 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:34.988031 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Aug 04 15:10:35.001459 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.001441 2559 log.go:25] "Validated CRI v1 runtime API" Aug 04 15:10:35.003416 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.003403 2559 log.go:25] "Validated CRI v1 image API" Aug 04 15:10:35.004146 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.004130 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Aug 04 15:10:35.006613 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.006594 2559 fs.go:135] Filesystem UUIDs: map[506d9b65-e97c-4381-89cd-f7fad4d688b5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8101c0ac-aabd-4c46-b9c1-e6619fef6c68:/dev/nvme0n1p4] Aug 04 15:10:35.006671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.006613 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Aug 04 15:10:35.012785 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.012679 2559 manager.go:216] Machine: {Timestamp:2026-08-04 15:10:35.011374889 +0000 UTC m=+0.365462638 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099733 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f6851fa0b710ac1ae59751e578c5 SystemUUID:ec28f685-1fa0-b710-ac1a-e59751e578c5 BootID:1607f246-a0dc-4f84-81ef-ae825fef6091 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:fb:c5:07:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:fb:c5:07:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:b7:c4:1a:b8:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Aug 04 15:10:35.012785 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.012784 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Aug 04 15:10:35.012890 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.012871 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Aug 04 15:10:35.014464 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.014448 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 15:10:35.015733 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.015706 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 04 15:10:35.015878 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.015736 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Aug 04 15:10:35.016599 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.016589 2559 topology_manager.go:138] "Creating topology manager with none policy" Aug 04 15:10:35.016633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.016602 2559 container_manager_linux.go:309] "Creating device plugin manager" Aug 04 15:10:35.016633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.016610 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 15:10:35.017258 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.017248 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Aug 04 15:10:35.018074 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.018064 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Aug 04 15:10:35.022329 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022317 2559 state_mem.go:36] "Initialized new in-memory state store" Aug 04 15:10:35.022443 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022433 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Aug 04 15:10:35.022487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022463 2559 kubelet.go:486] "Attempting to sync node with API server" Aug 04 15:10:35.022487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022472 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Aug 04 15:10:35.022487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022483 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Aug 04 15:10:35.022614 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022492 2559 kubelet.go:398] "Adding apiserver pod source" Aug 04 15:10:35.022614 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022501 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 04 15:10:35.022876 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022865 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 15:10:35.022911 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.022883 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Aug 04 15:10:35.025661 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.025639 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Aug 04 15:10:35.027169 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.027155 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Aug 04 15:10:35.027247 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.027185 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Aug 04 15:10:35.028288 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028277 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028293 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028299 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028305 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028317 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028326 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028337 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028342 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Aug 04 15:10:35.028346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028350 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Aug 04 15:10:35.028603 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028356 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Aug 04 15:10:35.028603 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028375 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Aug 04 15:10:35.028603 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.028381 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Aug 04 15:10:35.029078 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.029069 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Aug 04 15:10:35.029078 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.029078 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Aug 04 15:10:35.032192 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.032177 2559 server.go:1268] "Started kubelet" Aug 04 15:10:35.032275 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.032241 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Aug 04 15:10:35.032405 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.032354 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Aug 04 15:10:35.032500 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.032419 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Aug 04 15:10:35.033477 ip-10-0-138-35 systemd[1]: Started Kubernetes Kubelet. Aug 04 15:10:35.034169 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.034151 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Aug 04 15:10:35.041293 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.041268 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-35.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Aug 04 15:10:35.041384 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.041302 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Aug 04 15:10:35.041490 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.041464 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Aug 04 15:10:35.041662 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.041642 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Aug 04 15:10:35.041828 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.041811 2559 server.go:310] "Adding debug handlers to kubelet server" Aug 04 15:10:35.044667 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.043722 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8ad01ffa3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.032149923 +0000 UTC m=+0.386237674,LastTimestamp:2026-08-04 15:10:35.032149923 +0000 UTC m=+0.386237674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.045125 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045103 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 04 15:10:35.045125 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045111 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Aug 04 15:10:35.045264 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.045217 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.045825 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045792 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Aug 04 15:10:35.045825 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045827 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Aug 04 15:10:35.045978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045800 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Aug 04 15:10:35.045978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045946 2559 reconstruct.go:97] "Volume reconstruction finished" Aug 04 15:10:35.045978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045956 2559 reconciler.go:29] "Reconciler: start to sync state" Aug 04 15:10:35.046116 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.045995 2559 factory.go:55] Registering systemd factory Aug 04 15:10:35.046116 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.046010 2559 factory.go:223] Registration of the systemd container factory successfully Aug 04 15:10:35.046233 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.046220 2559 factory.go:153] Registering CRI-O factory Aug 04 15:10:35.046287 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.046237 2559 factory.go:223] Registration of the crio container factory successfully Aug 04 15:10:35.046340 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.046287 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Aug 04 15:10:35.046340 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.046310 2559 factory.go:103] Registering Raw factory Aug 04 15:10:35.046340 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.046324 2559 manager.go:1196] Started watching for new ooms in manager Aug 04 15:10:35.046780 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.046742 2559 manager.go:349] Starting recovery of all containers Aug 04 15:10:35.054082 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.054034 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Aug 04 15:10:35.054187 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.054094 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Aug 04 15:10:35.055025 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.055011 2559 manager.go:354] Recovery completed Aug 04 15:10:35.059389 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.059377 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.062103 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.062084 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.062193 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.062118 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.062193 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.062134 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.062838 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.062793 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Aug 04 15:10:35.062838 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.062807 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Aug 04 15:10:35.062838 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.062824 2559 state_mem.go:36] "Initialized new in-memory state store" Aug 04 15:10:35.062838 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.062830 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.063503 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.063489 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.063557 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.063514 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.063557 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.063526 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.064741 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.064650 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb073d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062101821 +0000 UTC m=+0.416189574,LastTimestamp:2026-08-04 15:10:35.062101821 +0000 UTC m=+0.416189574,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.065807 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.065794 2559 policy_none.go:49] "None policy: Start" Aug 04 15:10:35.065881 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.065814 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Aug 04 15:10:35.065881 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.065828 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Aug 04 15:10:35.067092 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.067080 2559 policy_none.go:47] "Start" Aug 04 15:10:35.072696 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.072626 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb633f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062125375 +0000 UTC m=+0.416213129,LastTimestamp:2026-08-04 15:10:35.062125375 +0000 UTC m=+0.416213129,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.080194 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.079617 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb98a5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062139045 +0000 UTC m=+0.416226797,LastTimestamp:2026-08-04 15:10:35.062139045 +0000 UTC m=+0.416226797,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.081595 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.081564 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfl7k" Aug 04 15:10:35.083244 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.083185 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-35.ec2.internal.18c8a2c8aecb073d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb073d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062101821 +0000 UTC m=+0.416189574,LastTimestamp:2026-08-04 15:10:35.063502599 +0000 UTC m=+0.417590348,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.093419 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.093344 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-35.ec2.internal.18c8a2c8aecb633f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb633f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062125375 +0000 UTC m=+0.416213129,LastTimestamp:2026-08-04 15:10:35.063518943 +0000 UTC m=+0.417606693,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.094613 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.094597 2559 manager.go:337] "Starting Device Plugin manager" Aug 04 15:10:35.094701 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.094636 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Aug 04 15:10:35.094701 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.094649 2559 server.go:87] "Starting device plugin registration server" Aug 04 15:10:35.094907 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.094894 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Aug 04 15:10:35.094949 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.094910 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Aug 04 15:10:35.095033 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.095020 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Aug 04 15:10:35.095113 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.095094 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Aug 04 15:10:35.095113 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.095104 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 04 15:10:35.095606 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.095585 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Aug 04 15:10:35.095674 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.095636 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.096898 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.096836 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-35.ec2.internal.18c8a2c8aecb98a5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb98a5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062139045 +0000 UTC m=+0.416226797,LastTimestamp:2026-08-04 15:10:35.063531666 +0000 UTC m=+0.417619632,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.098238 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.098221 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sfl7k" Aug 04 15:10:35.112004 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.111946 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8b19646ad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.108976301 +0000 UTC m=+0.463064042,LastTimestamp:2026-08-04 15:10:35.108976301 +0000 UTC m=+0.463064042,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.195307 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.195269 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.196270 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.196253 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.196353 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.196284 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.196353 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.196297 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.196353 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.196323 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.209152 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.209040 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-35.ec2.internal.18c8a2c8aecb073d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb073d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062101821 +0000 UTC m=+0.416189574,LastTimestamp:2026-08-04 15:10:35.196270083 +0000 UTC m=+0.550357833,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.221428 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.221405 2559 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.221538 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.221392 2559 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-35.ec2.internal.18c8a2c8aecb633f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-35.ec2.internal.18c8a2c8aecb633f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-35.ec2.internal,UID:ip-10-0-138-35.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-35.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-35.ec2.internal,},FirstTimestamp:2026-08-04 15:10:35.062125375 +0000 UTC m=+0.416213129,LastTimestamp:2026-08-04 15:10:35.196290654 +0000 UTC m=+0.550378404,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-35.ec2.internal,}" Aug 04 15:10:35.266201 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.266173 2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-35.ec2.internal\" not found" node="ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.280599 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.280559 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Aug 04 15:10:35.281696 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.281678 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Aug 04 15:10:35.281783 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.281715 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Aug 04 15:10:35.281783 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.281734 2559 kubelet.go:2448] "Starting kubelet main sync loop" Aug 04 15:10:35.281783 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.281769 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Aug 04 15:10:35.284895 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.284873 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:10:35.382133 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.382097 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"] Aug 04 15:10:35.382275 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.382176 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.383129 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.383112 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.383182 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.383143 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.383182 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.383156 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.384418 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.384406 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.384607 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.384592 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.384648 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.384625 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.385224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.385208 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.385319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.385226 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.385319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.385251 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.385319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.385263 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.385319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.385233 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.385319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.385310 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.386405 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.386392 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.386457 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.386415 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.387175 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.387159 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.387277 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.387188 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.387277 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.387198 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.415323 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.415299 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-35.ec2.internal\" not found" node="ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.419712 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.419696 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-35.ec2.internal\" not found" node="ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.421737 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.421723 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:35.422513 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.422495 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:35.422597 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.422527 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:35.422597 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.422543 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:35.422597 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.422564 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.441649 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.441629 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.441728 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.441653 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-35.ec2.internal\": node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.448007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.447986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747b5a69ee89bac7542c3f6d437b8cbc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"747b5a69ee89bac7542c3f6d437b8cbc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.448060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.448012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747b5a69ee89bac7542c3f6d437b8cbc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"747b5a69ee89bac7542c3f6d437b8cbc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.448060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.448030 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b643114c2bf06208c77f7bdab0056d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.ec2.internal\" (UID: \"14b643114c2bf06208c77f7bdab0056d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.513638 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.513549 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.549143 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.549115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747b5a69ee89bac7542c3f6d437b8cbc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"747b5a69ee89bac7542c3f6d437b8cbc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.549233 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.549147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747b5a69ee89bac7542c3f6d437b8cbc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"747b5a69ee89bac7542c3f6d437b8cbc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.549233 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.549165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b643114c2bf06208c77f7bdab0056d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.ec2.internal\" (UID: \"14b643114c2bf06208c77f7bdab0056d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.549331 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.549242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747b5a69ee89bac7542c3f6d437b8cbc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"747b5a69ee89bac7542c3f6d437b8cbc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.549331 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.549296 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/14b643114c2bf06208c77f7bdab0056d-config\") pod \"kube-apiserver-proxy-ip-10-0-138-35.ec2.internal\" (UID: \"14b643114c2bf06208c77f7bdab0056d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.549404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.549341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747b5a69ee89bac7542c3f6d437b8cbc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal\" (UID: \"747b5a69ee89bac7542c3f6d437b8cbc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.614534 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.614493 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.714878 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.714835 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.718221 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.718200 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.723235 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.722971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" Aug 04 15:10:35.817798 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.817700 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.923218 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:35.923170 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:35.979815 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.979783 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Aug 04 15:10:35.980323 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:35.979974 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Aug 04 15:10:36.025500 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.025473 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.120498 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.120411 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Aug 04 15:10:36.121396 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.121380 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientMemory" Aug 04 15:10:36.121487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.121416 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasNoDiskPressure" Aug 04 15:10:36.121487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.121431 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeHasSufficientPID" Aug 04 15:10:36.125945 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.125929 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.228126 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.228098 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.228236 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.228116 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 15:05:35 +0000 UTC" deadline="2028-03-14 12:16:48.081420287 +0000 UTC" Aug 04 15:10:36.228236 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.228152 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14109h6m11.85327206s" Aug 04 15:10:36.328656 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.328620 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.363633 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:36.363572 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b643114c2bf06208c77f7bdab0056d.slice/crio-b3543af388e2711c23b6b1b66ee935618a6e6dd6812de89c72eff83abb4ce265 WatchSource:0}: Error finding container b3543af388e2711c23b6b1b66ee935618a6e6dd6812de89c72eff83abb4ce265: Status 404 returned error can't find the container with id b3543af388e2711c23b6b1b66ee935618a6e6dd6812de89c72eff83abb4ce265 Aug 04 15:10:36.368550 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.368533 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:10:36.429554 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.429519 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.470825 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.470793 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:10:36.480490 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.480465 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:10:36.529799 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.529763 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.636261 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.636179 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.736993 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.736947 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.837740 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.837702 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.857951 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:36.857916 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747b5a69ee89bac7542c3f6d437b8cbc.slice/crio-1e9bb5c5f6890a8ae135929bf808bd0000faacb2a03ec4d36c1be78c86a44b22 WatchSource:0}: Error finding container 1e9bb5c5f6890a8ae135929bf808bd0000faacb2a03ec4d36c1be78c86a44b22: Status 404 returned error can't find the container with id 1e9bb5c5f6890a8ae135929bf808bd0000faacb2a03ec4d36c1be78c86a44b22 Aug 04 15:10:36.942621 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:36.942503 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-35.ec2.internal\" not found" Aug 04 15:10:36.984554 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:36.984399 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:10:37.026367 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.026330 2559 apiserver.go:52] "Watching apiserver" Aug 04 15:10:37.038077 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.038041 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Aug 04 15:10:37.039439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.039344 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6nqlc","kube-system/konnectivity-agent-4nfzb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7","openshift-dns/node-resolver-j8h9m","openshift-image-registry/node-ca-lk76r","openshift-multus/multus-additional-cni-plugins-vk2fg","openshift-multus/multus-wzhmd","openshift-multus/network-metrics-daemon-w2npp","openshift-cluster-node-tuning-operator/tuned-c7kw7","openshift-network-diagnostics/network-check-target-kl72z","openshift-network-operator/iptables-alerter-btv2r","openshift-ovn-kubernetes/ovnkube-node-646lg"] Aug 04 15:10:37.041331 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.041303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.042881 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.042857 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.044609 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.044572 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.046474 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.045785 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" Aug 04 15:10:37.046474 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.045803 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Aug 04 15:10:37.046474 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.045910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.046960 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.046863 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Aug 04 15:10:37.047079 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047064 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Aug 04 15:10:37.047196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Aug 04 15:10:37.047262 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047243 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Aug 04 15:10:37.047325 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047287 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tlqc2\"" Aug 04 15:10:37.047399 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047377 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hwc6n\"" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047648 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.047723 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047885 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047935 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047995 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.047935 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Aug 04 15:10:37.048141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.048120 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Aug 04 15:10:37.048532 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.048200 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-894nd\"" Aug 04 15:10:37.048532 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.048208 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Aug 04 15:10:37.048986 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.048922 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5q6x7\"" Aug 04 15:10:37.049575 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.049235 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.050701 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.050681 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:37.050796 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.050755 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:37.052010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.051848 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:10:37.052010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.051907 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gwmsq\"" Aug 04 15:10:37.052153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.052126 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Aug 04 15:10:37.052938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.052910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.053024 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.052983 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:37.054325 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.054305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.055609 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.055575 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.056266 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:37.056372 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.056372 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-multus-certs\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.056372 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056345 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-etc-kubernetes\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.056539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-hosts-file\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.056539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa1d177c-d3ac-425d-a836-58dc1dc7b425-serviceca\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.056539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.056539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056468 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpxbd\" (UniqueName: \"kubernetes.io/projected/1f34acf7-da6c-4fb9-aad4-dff78830a97e-kube-api-access-lpxbd\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.056539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-netns\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.056539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-daemon-config\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056544 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cnibin\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056570 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-tmp-dir\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa1d177c-d3ac-425d-a836-58dc1dc7b425-host\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056640 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-hostroot\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056664 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjg6\" (UniqueName: \"kubernetes.io/projected/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-kube-api-access-ckjg6\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnpcm\" (UniqueName: \"kubernetes.io/projected/fa1d177c-d3ac-425d-a836-58dc1dc7b425-kube-api-access-jnpcm\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056722 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056773 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-os-release\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.056863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-k8s-cni-cncf-io\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.057407 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.056882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-cni-multus\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.057495 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-conf-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.057495 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-cnibin\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.057495 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-cni-binary-copy\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057498 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92qd8\" (UniqueName: \"kubernetes.io/projected/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-kube-api-access-92qd8\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057524 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-system-cni-dir\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057547 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-system-cni-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057596 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-os-release\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-cni-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057635 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Aug 04 15:10:37.057682 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-cni-bin\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.058095 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057692 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-kubelet\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.058095 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz58h\" (UniqueName: \"kubernetes.io/projected/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-kube-api-access-wz58h\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.058095 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-socket-dir-parent\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.058095 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.057839 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Aug 04 15:10:37.058095 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.058019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.058410 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.058215 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k844d\"" Aug 04 15:10:37.058560 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.058540 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Aug 04 15:10:37.058651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.058561 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9kqz9\"" Aug 04 15:10:37.058651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.058624 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Aug 04 15:10:37.059008 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.058991 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Aug 04 15:10:37.059840 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.059819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.063414 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.063385 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:10:37.064106 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.064084 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Aug 04 15:10:37.064576 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.064325 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8445m\"" Aug 04 15:10:37.064576 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.064442 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Aug 04 15:10:37.064744 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.064665 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Aug 04 15:10:37.064797 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.064778 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Aug 04 15:10:37.066773 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.066693 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Aug 04 15:10:37.067105 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.067084 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Aug 04 15:10:37.067195 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.067120 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Aug 04 15:10:37.067805 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.067567 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l7zmw\"" Aug 04 15:10:37.067805 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.067627 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Aug 04 15:10:37.068305 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.068287 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 15:10:37.068402 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.068387 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" Aug 04 15:10:37.068657 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.068635 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Aug 04 15:10:37.068750 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.068703 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal"] Aug 04 15:10:37.081386 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.081365 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Aug 04 15:10:37.081619 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.081334 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal"] Aug 04 15:10:37.114984 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.114951 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vsnst" Aug 04 15:10:37.126206 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.126180 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vsnst" Aug 04 15:10:37.147410 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.147302 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Aug 04 15:10:37.157967 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.157938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-socket-dir-parent\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.158137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.157991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-var-lib-kubelet\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.158137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158020 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-device-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.158137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158050 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-var-lib-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.158137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158078 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-etc-kubernetes\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.158137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-tmp\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.158137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158122 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-hosts-file\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa1d177c-d3ac-425d-a836-58dc1dc7b425-serviceca\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-socket-dir-parent\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-etc-kubernetes\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpxbd\" (UniqueName: \"kubernetes.io/projected/1f34acf7-da6c-4fb9-aad4-dff78830a97e-kube-api-access-lpxbd\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158223 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-hosts-file\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-sys-fs\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz865\" (UniqueName: \"kubernetes.io/projected/39ce0ab6-9006-4378-b717-9cae3c2e27ee-kube-api-access-zz865\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-systemd\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-run\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158433 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-sys\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158455 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-etc-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158502 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.158564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-ovnkube-config\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa1d177c-d3ac-425d-a836-58dc1dc7b425-host\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158637 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a1d41dd-149a-4de6-a3e7-993833e83ac3-agent-certs\") pod \"konnectivity-agent-4nfzb\" (UID: \"0a1d41dd-149a-4de6-a3e7-993833e83ac3\") " pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158648 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fa1d177c-d3ac-425d-a836-58dc1dc7b425-host\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-node-log\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fa1d177c-d3ac-425d-a836-58dc1dc7b425-serviceca\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e629fc59-eefc-49cd-9bb5-9a329b04633b-kubelet-config\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnpcm\" (UniqueName: \"kubernetes.io/projected/fa1d177c-d3ac-425d-a836-58dc1dc7b425-kube-api-access-jnpcm\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158791 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-cni-multus\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-conf-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-tuned\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-registration-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.158976 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-cnibin\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92qd8\" (UniqueName: \"kubernetes.io/projected/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-kube-api-access-92qd8\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-modprobe-d\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.159334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysctl-conf\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zdk\" (UniqueName: \"kubernetes.io/projected/4506fb2a-2d1b-4df3-b915-682295356ecd-kube-api-access-r5zdk\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39ce0ab6-9006-4378-b717-9cae3c2e27ee-iptables-alerter-script\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159129 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-ovn\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159152 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-ovnkube-script-lib\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-host\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz58h\" (UniqueName: \"kubernetes.io/projected/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-kube-api-access-wz58h\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39ce0ab6-9006-4378-b717-9cae3c2e27ee-host-slash\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159304 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-cni-multus\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-run-netns\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-conf-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-multus-certs\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysctl-d\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.159437 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:37.159944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159509 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-cnibin\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.159521 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs podName:d26e3571-af26-4a3d-ab3f-0800c3b1f66b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:37.659489111 +0000 UTC m=+3.013576854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs") pod "network-metrics-daemon-w2npp" (UID: "d26e3571-af26-4a3d-ab3f-0800c3b1f66b") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-multus-certs\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/194c9b51-0bac-4104-8404-0437944a00f4-ovn-node-metrics-cert\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysconfig\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-netns\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159758 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-daemon-config\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159773 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-lib-modules\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jv7b\" (UniqueName: \"kubernetes.io/projected/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-kube-api-access-4jv7b\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e629fc59-eefc-49cd-9bb5-9a329b04633b-dbus\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-socket-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159855 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-etc-selinux\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159882 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cnibin\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-tmp-dir\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.160524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-hostroot\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-hostroot\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjg6\" (UniqueName: \"kubernetes.io/projected/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-kube-api-access-ckjg6\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.159978 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-kubelet\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-slash\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-log-socket\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-cni-netd\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-os-release\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-k8s-cni-cncf-io\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-systemd-units\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160119 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-cni-bin\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-env-overrides\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160159 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a1d41dd-149a-4de6-a3e7-993833e83ac3-konnectivity-ca\") pod \"konnectivity-agent-4nfzb\" (UID: \"0a1d41dd-149a-4de6-a3e7-993833e83ac3\") " pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160188 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-cni-binary-copy\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cnibin\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-system-cni-dir\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.161351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160238 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160254 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-system-cni-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160287 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjlc\" (UniqueName: \"kubernetes.io/projected/194c9b51-0bac-4104-8404-0437944a00f4-kube-api-access-xdjlc\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-os-release\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-cni-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-cni-bin\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-kubelet\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-kubernetes\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160431 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-systemd\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160438 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-tmp-dir\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160522 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-os-release\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160579 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-system-cni-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-os-release\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-daemon-config\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-kubelet\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160752 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-var-lib-cni-bin\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160117 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-netns\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160787 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-multus-cni-dir\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162958 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-host-run-k8s-cni-cncf-io\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.162958 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160839 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1f34acf7-da6c-4fb9-aad4-dff78830a97e-system-cni-dir\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.162958 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.160907 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1f34acf7-da6c-4fb9-aad4-dff78830a97e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.162958 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.161143 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-cni-binary-copy\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.177252 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.177219 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Aug 04 15:10:37.181371 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.181317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjg6\" (UniqueName: \"kubernetes.io/projected/ab0ee247-651d-4fd2-a2ad-af1ede75d32e-kube-api-access-ckjg6\") pod \"multus-wzhmd\" (UID: \"ab0ee247-651d-4fd2-a2ad-af1ede75d32e\") " pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.181503 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.181401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92qd8\" (UniqueName: \"kubernetes.io/projected/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-kube-api-access-92qd8\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:37.181579 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.181557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz58h\" (UniqueName: \"kubernetes.io/projected/7632fc34-ddbf-4d83-8048-0a9d86cdf1a0-kube-api-access-wz58h\") pod \"node-resolver-j8h9m\" (UID: \"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0\") " pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.181707 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.181589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpxbd\" (UniqueName: \"kubernetes.io/projected/1f34acf7-da6c-4fb9-aad4-dff78830a97e-kube-api-access-lpxbd\") pod \"multus-additional-cni-plugins-vk2fg\" (UID: \"1f34acf7-da6c-4fb9-aad4-dff78830a97e\") " pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.181707 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.181690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnpcm\" (UniqueName: \"kubernetes.io/projected/fa1d177c-d3ac-425d-a836-58dc1dc7b425-kube-api-access-jnpcm\") pod \"node-ca-lk76r\" (UID: \"fa1d177c-d3ac-425d-a836-58dc1dc7b425\") " pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.242508 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.242425 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Aug 04 15:10:37.260784 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-sys-fs\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.260945 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260792 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz865\" (UniqueName: \"kubernetes.io/projected/39ce0ab6-9006-4378-b717-9cae3c2e27ee-kube-api-access-zz865\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.260945 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-systemd\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.260945 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260875 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-systemd\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.260945 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-sys-fs\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-run\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-sys\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.260999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-etc-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-run\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-ovnkube-config\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261136 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-sys\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a1d41dd-149a-4de6-a3e7-993833e83ac3-agent-certs\") pod \"konnectivity-agent-4nfzb\" (UID: \"0a1d41dd-149a-4de6-a3e7-993833e83ac3\") " pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261173 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-node-log\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261180 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-etc-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261183 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-run-ovn-kubernetes\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261257 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e629fc59-eefc-49cd-9bb5-9a329b04633b-kubelet-config\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261312 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-node-log\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e629fc59-eefc-49cd-9bb5-9a329b04633b-kubelet-config\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-tuned\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-registration-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.261507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-modprobe-d\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysctl-conf\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zdk\" (UniqueName: \"kubernetes.io/projected/4506fb2a-2d1b-4df3-b915-682295356ecd-kube-api-access-r5zdk\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39ce0ab6-9006-4378-b717-9cae3c2e27ee-iptables-alerter-script\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-ovn\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261658 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-ovnkube-script-lib\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-host\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261709 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39ce0ab6-9006-4378-b717-9cae3c2e27ee-host-slash\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-run-netns\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-modprobe-d\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysctl-d\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261843 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-registration-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/194c9b51-0bac-4104-8404-0437944a00f4-ovn-node-metrics-cert\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysconfig\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-lib-modules\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.262128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jv7b\" (UniqueName: \"kubernetes.io/projected/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-kube-api-access-4jv7b\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e629fc59-eefc-49cd-9bb5-9a329b04633b-dbus\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.261983 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-socket-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-etc-selinux\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-kubelet\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262059 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-ovnkube-config\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-slash\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262083 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysctl-conf\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-log-socket\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262119 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-run-ovn\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-cni-netd\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-systemd-units\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262180 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-cni-bin\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262204 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-env-overrides\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a1d41dd-149a-4de6-a3e7-993833e83ac3-konnectivity-ca\") pod \"konnectivity-agent-4nfzb\" (UID: \"0a1d41dd-149a-4de6-a3e7-993833e83ac3\") " pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjlc\" (UniqueName: \"kubernetes.io/projected/194c9b51-0bac-4104-8404-0437944a00f4-kube-api-access-xdjlc\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-kubernetes\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-systemd\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-systemd\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-kubelet\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-etc-selinux\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262493 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-slash\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262506 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-var-lib-kubelet\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-device-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262561 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-ovnkube-script-lib\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262538 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-log-socket\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-var-lib-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-var-lib-openvswitch\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262608 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-cni-netd\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-tmp\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-run-netns\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.262730 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.262785 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret podName:e629fc59-eefc-49cd-9bb5-9a329b04633b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:37.762767852 +0000 UTC m=+3.116855591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret") pod "global-pull-secret-syncer-6nqlc" (UID: "e629fc59-eefc-49cd-9bb5-9a329b04633b") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:37.263872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/39ce0ab6-9006-4378-b717-9cae3c2e27ee-iptables-alerter-script\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262890 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e629fc59-eefc-49cd-9bb5-9a329b04633b-dbus\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262985 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/194c9b51-0bac-4104-8404-0437944a00f4-env-overrides\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-socket-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263051 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/39ce0ab6-9006-4378-b717-9cae3c2e27ee-host-slash\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263101 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-device-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263148 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-systemd-units\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-cni-bin\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.262172 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-host\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a1d41dd-149a-4de6-a3e7-993833e83ac3-konnectivity-ca\") pod \"konnectivity-agent-4nfzb\" (UID: \"0a1d41dd-149a-4de6-a3e7-993833e83ac3\") " pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-var-lib-kubelet\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4506fb2a-2d1b-4df3-b915-682295356ecd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysctl-d\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-kubernetes\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-lib-modules\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263409 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-sysconfig\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.263411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/194c9b51-0bac-4104-8404-0437944a00f4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.264688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.264310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a1d41dd-149a-4de6-a3e7-993833e83ac3-agent-certs\") pod \"konnectivity-agent-4nfzb\" (UID: \"0a1d41dd-149a-4de6-a3e7-993833e83ac3\") " pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.265463 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.264922 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-etc-tuned\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.265830 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.265808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/194c9b51-0bac-4104-8404-0437944a00f4-ovn-node-metrics-cert\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.269695 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.266427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-tmp\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.271541 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.271264 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:10:37.271541 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.271291 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:10:37.271541 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.271306 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bbcld for pod openshift-network-diagnostics/network-check-target-kl72z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:37.271541 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.271370 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld podName:86dfd1b4-818f-4c5f-b7da-63998cbf15d8 nodeName:}" failed. No retries permitted until 2026-08-04 15:10:37.771351793 +0000 UTC m=+3.125439532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bbcld" (UniqueName: "kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld") pod "network-check-target-kl72z" (UID: "86dfd1b4-818f-4c5f-b7da-63998cbf15d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:37.273531 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.273508 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zdk\" (UniqueName: \"kubernetes.io/projected/4506fb2a-2d1b-4df3-b915-682295356ecd-kube-api-access-r5zdk\") pod \"aws-ebs-csi-driver-node-krsk7\" (UID: \"4506fb2a-2d1b-4df3-b915-682295356ecd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.274707 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.274683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz865\" (UniqueName: \"kubernetes.io/projected/39ce0ab6-9006-4378-b717-9cae3c2e27ee-kube-api-access-zz865\") pod \"iptables-alerter-btv2r\" (UID: \"39ce0ab6-9006-4378-b717-9cae3c2e27ee\") " pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.275010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.274957 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjlc\" (UniqueName: \"kubernetes.io/projected/194c9b51-0bac-4104-8404-0437944a00f4-kube-api-access-xdjlc\") pod \"ovnkube-node-646lg\" (UID: \"194c9b51-0bac-4104-8404-0437944a00f4\") " pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.276076 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.276058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jv7b\" (UniqueName: \"kubernetes.io/projected/d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f-kube-api-access-4jv7b\") pod \"tuned-c7kw7\" (UID: \"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f\") " pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.286452 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.286389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" event={"ID":"747b5a69ee89bac7542c3f6d437b8cbc","Type":"ContainerStarted","Data":"1e9bb5c5f6890a8ae135929bf808bd0000faacb2a03ec4d36c1be78c86a44b22"} Aug 04 15:10:37.287683 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.287659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" event={"ID":"14b643114c2bf06208c77f7bdab0056d","Type":"ContainerStarted","Data":"b3543af388e2711c23b6b1b66ee935618a6e6dd6812de89c72eff83abb4ce265"} Aug 04 15:10:37.355869 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.355838 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j8h9m" Aug 04 15:10:37.364610 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.364581 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7632fc34_ddbf_4d83_8048_0a9d86cdf1a0.slice/crio-69a9bc7cebe904caa9347d1408318ed5ad208fc769c7c0f14d31440f11161497 WatchSource:0}: Error finding container 69a9bc7cebe904caa9347d1408318ed5ad208fc769c7c0f14d31440f11161497: Status 404 returned error can't find the container with id 69a9bc7cebe904caa9347d1408318ed5ad208fc769c7c0f14d31440f11161497 Aug 04 15:10:37.368060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.368020 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lk76r" Aug 04 15:10:37.376378 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.376349 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1d177c_d3ac_425d_a836_58dc1dc7b425.slice/crio-89f19db6316ffc66ef3daac310fc9d215647730b229a353742ed92a80e0e2a9f WatchSource:0}: Error finding container 89f19db6316ffc66ef3daac310fc9d215647730b229a353742ed92a80e0e2a9f: Status 404 returned error can't find the container with id 89f19db6316ffc66ef3daac310fc9d215647730b229a353742ed92a80e0e2a9f Aug 04 15:10:37.379533 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.379514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" Aug 04 15:10:37.385638 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.385612 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wzhmd" Aug 04 15:10:37.386678 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.386653 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f34acf7_da6c_4fb9_aad4_dff78830a97e.slice/crio-4c40c53943620576b41a0ab4042d2d94dad375a8419b503da6920e6e54f04881 WatchSource:0}: Error finding container 4c40c53943620576b41a0ab4042d2d94dad375a8419b503da6920e6e54f04881: Status 404 returned error can't find the container with id 4c40c53943620576b41a0ab4042d2d94dad375a8419b503da6920e6e54f04881 Aug 04 15:10:37.392831 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.392808 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0ee247_651d_4fd2_a2ad_af1ede75d32e.slice/crio-8c79e0f4e60272f072d38c01c9d115ce0510fae9c07380a1503a627bcd88f5f6 WatchSource:0}: Error finding container 8c79e0f4e60272f072d38c01c9d115ce0510fae9c07380a1503a627bcd88f5f6: Status 404 returned error can't find the container with id 8c79e0f4e60272f072d38c01c9d115ce0510fae9c07380a1503a627bcd88f5f6 Aug 04 15:10:37.393430 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.393400 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" Aug 04 15:10:37.400991 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.400962 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ed10dc_85d2_4dd0_9273_11e4e3d4f77f.slice/crio-85262b8b38b69c3d80020d686eeae4b09aa2b826ae580627db18c3201209423a WatchSource:0}: Error finding container 85262b8b38b69c3d80020d686eeae4b09aa2b826ae580627db18c3201209423a: Status 404 returned error can't find the container with id 85262b8b38b69c3d80020d686eeae4b09aa2b826ae580627db18c3201209423a Aug 04 15:10:37.402066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.402048 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:37.410660 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.410634 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" Aug 04 15:10:37.411242 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.411115 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1d41dd_149a_4de6_a3e7_993833e83ac3.slice/crio-2b8943911e4a9ca20c1936506c8610cacdde0d630e895c63206b49c07a5a0637 WatchSource:0}: Error finding container 2b8943911e4a9ca20c1936506c8610cacdde0d630e895c63206b49c07a5a0637: Status 404 returned error can't find the container with id 2b8943911e4a9ca20c1936506c8610cacdde0d630e895c63206b49c07a5a0637 Aug 04 15:10:37.419858 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.419835 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-btv2r" Aug 04 15:10:37.420052 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.420015 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4506fb2a_2d1b_4df3_b915_682295356ecd.slice/crio-900cdc3844d17c9e39fbf184160e727dbbb444da8b53274746cc09110d5fcf9c WatchSource:0}: Error finding container 900cdc3844d17c9e39fbf184160e727dbbb444da8b53274746cc09110d5fcf9c: Status 404 returned error can't find the container with id 900cdc3844d17c9e39fbf184160e727dbbb444da8b53274746cc09110d5fcf9c Aug 04 15:10:37.427020 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.426995 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:10:37.430407 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.430380 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ce0ab6_9006_4378_b717_9cae3c2e27ee.slice/crio-ca2cbbf9f5c0d504cf44bdf321080e99dad4932db1a73dca2b5b79f9cd8d6d59 WatchSource:0}: Error finding container ca2cbbf9f5c0d504cf44bdf321080e99dad4932db1a73dca2b5b79f9cd8d6d59: Status 404 returned error can't find the container with id ca2cbbf9f5c0d504cf44bdf321080e99dad4932db1a73dca2b5b79f9cd8d6d59 Aug 04 15:10:37.435560 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:10:37.435535 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod194c9b51_0bac_4104_8404_0437944a00f4.slice/crio-fc69ea6e2c21e44c92e3751a67011ceee39a293e261c5dead47d1bf2f569867a WatchSource:0}: Error finding container fc69ea6e2c21e44c92e3751a67011ceee39a293e261c5dead47d1bf2f569867a: Status 404 returned error can't find the container with id fc69ea6e2c21e44c92e3751a67011ceee39a293e261c5dead47d1bf2f569867a Aug 04 15:10:37.666065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.665983 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:37.666234 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.666158 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:37.666290 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.666238 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs podName:d26e3571-af26-4a3d-ab3f-0800c3b1f66b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:38.666213843 +0000 UTC m=+4.020301588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs") pod "network-metrics-daemon-w2npp" (UID: "d26e3571-af26-4a3d-ab3f-0800c3b1f66b") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:37.767344 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.767309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:37.767525 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.767495 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:37.767626 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.767574 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret podName:e629fc59-eefc-49cd-9bb5-9a329b04633b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:38.767552259 +0000 UTC m=+4.121640006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret") pod "global-pull-secret-syncer-6nqlc" (UID: "e629fc59-eefc-49cd-9bb5-9a329b04633b") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:37.868054 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:37.868021 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:37.868238 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.868208 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:10:37.868238 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.868227 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:10:37.868346 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.868241 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bbcld for pod openshift-network-diagnostics/network-check-target-kl72z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:37.868346 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:37.868312 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld podName:86dfd1b4-818f-4c5f-b7da-63998cbf15d8 nodeName:}" failed. No retries permitted until 2026-08-04 15:10:38.868290855 +0000 UTC m=+4.222378616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bbcld" (UniqueName: "kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld") pod "network-check-target-kl72z" (UID: "86dfd1b4-818f-4c5f-b7da-63998cbf15d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:38.129302 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.129215 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:05:37 +0000 UTC" deadline="2028-04-01 18:10:23.738293557 +0000 UTC" Aug 04 15:10:38.129302 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.129254 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14546h59m45.609044602s" Aug 04 15:10:38.283031 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.282898 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:38.283408 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.283376 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:38.291708 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.291672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"fc69ea6e2c21e44c92e3751a67011ceee39a293e261c5dead47d1bf2f569867a"} Aug 04 15:10:38.293065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.292965 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-btv2r" event={"ID":"39ce0ab6-9006-4378-b717-9cae3c2e27ee","Type":"ContainerStarted","Data":"ca2cbbf9f5c0d504cf44bdf321080e99dad4932db1a73dca2b5b79f9cd8d6d59"} Aug 04 15:10:38.295010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.294935 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4nfzb" event={"ID":"0a1d41dd-149a-4de6-a3e7-993833e83ac3","Type":"ContainerStarted","Data":"2b8943911e4a9ca20c1936506c8610cacdde0d630e895c63206b49c07a5a0637"} Aug 04 15:10:38.296431 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.296404 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wzhmd" event={"ID":"ab0ee247-651d-4fd2-a2ad-af1ede75d32e","Type":"ContainerStarted","Data":"8c79e0f4e60272f072d38c01c9d115ce0510fae9c07380a1503a627bcd88f5f6"} Aug 04 15:10:38.297906 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.297879 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerStarted","Data":"4c40c53943620576b41a0ab4042d2d94dad375a8419b503da6920e6e54f04881"} Aug 04 15:10:38.299384 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.299359 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lk76r" event={"ID":"fa1d177c-d3ac-425d-a836-58dc1dc7b425","Type":"ContainerStarted","Data":"89f19db6316ffc66ef3daac310fc9d215647730b229a353742ed92a80e0e2a9f"} Aug 04 15:10:38.300802 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.300778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j8h9m" event={"ID":"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0","Type":"ContainerStarted","Data":"69a9bc7cebe904caa9347d1408318ed5ad208fc769c7c0f14d31440f11161497"} Aug 04 15:10:38.302496 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.302425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" event={"ID":"4506fb2a-2d1b-4df3-b915-682295356ecd","Type":"ContainerStarted","Data":"900cdc3844d17c9e39fbf184160e727dbbb444da8b53274746cc09110d5fcf9c"} Aug 04 15:10:38.304148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.303791 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" event={"ID":"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f","Type":"ContainerStarted","Data":"85262b8b38b69c3d80020d686eeae4b09aa2b826ae580627db18c3201209423a"} Aug 04 15:10:38.674535 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.674480 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:38.674756 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.674743 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:38.674838 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.674814 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs podName:d26e3571-af26-4a3d-ab3f-0800c3b1f66b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:40.674794454 +0000 UTC m=+6.028882214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs") pod "network-metrics-daemon-w2npp" (UID: "d26e3571-af26-4a3d-ab3f-0800c3b1f66b") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:38.775857 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.775816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:38.776053 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.776006 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:38.776121 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.776072 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret podName:e629fc59-eefc-49cd-9bb5-9a329b04633b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:40.776052334 +0000 UTC m=+6.130140082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret") pod "global-pull-secret-syncer-6nqlc" (UID: "e629fc59-eefc-49cd-9bb5-9a329b04633b") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:38.877917 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:38.877087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:38.877917 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.877316 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:10:38.877917 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.877338 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:10:38.877917 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.877367 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bbcld for pod openshift-network-diagnostics/network-check-target-kl72z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:38.877917 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:38.877427 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld podName:86dfd1b4-818f-4c5f-b7da-63998cbf15d8 nodeName:}" failed. No retries permitted until 2026-08-04 15:10:40.877407948 +0000 UTC m=+6.231495689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bbcld" (UniqueName: "kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld") pod "network-check-target-kl72z" (UID: "86dfd1b4-818f-4c5f-b7da-63998cbf15d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:39.129562 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:39.129466 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 15:05:37 +0000 UTC" deadline="2028-01-19 04:23:54.509565805 +0000 UTC" Aug 04 15:10:39.129562 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:39.129513 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12781h13m15.3800574s" Aug 04 15:10:39.282359 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:39.282144 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:39.282520 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:39.282466 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:39.282926 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:39.282907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:39.283063 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:39.283042 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:40.282554 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:40.282471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:40.283013 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.282641 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:40.325535 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:40.325496 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" event={"ID":"747b5a69ee89bac7542c3f6d437b8cbc","Type":"ContainerStarted","Data":"8f9d06867ddf6975c4baca14e1e39b81b1fd6dcf4859cefdf0a5ee13897a78c2"} Aug 04 15:10:40.328466 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:40.328437 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" event={"ID":"14b643114c2bf06208c77f7bdab0056d","Type":"ContainerStarted","Data":"4d48c31cc7515292c881e65d623d50be953d0c6256d460a23711f3694d590c95"} Aug 04 15:10:40.694998 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:40.694259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:40.694998 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.694461 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:40.694998 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.694524 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs podName:d26e3571-af26-4a3d-ab3f-0800c3b1f66b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:44.694505883 +0000 UTC m=+10.048593639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs") pod "network-metrics-daemon-w2npp" (UID: "d26e3571-af26-4a3d-ab3f-0800c3b1f66b") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:40.794790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:40.794752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:40.794983 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.794934 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:40.795059 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.794996 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret podName:e629fc59-eefc-49cd-9bb5-9a329b04633b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:44.794978414 +0000 UTC m=+10.149066159 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret") pod "global-pull-secret-syncer-6nqlc" (UID: "e629fc59-eefc-49cd-9bb5-9a329b04633b") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:40.895390 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:40.895354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:40.895560 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.895529 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:10:40.895560 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.895551 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:10:40.895690 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.895563 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bbcld for pod openshift-network-diagnostics/network-check-target-kl72z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:40.895690 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:40.895637 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld podName:86dfd1b4-818f-4c5f-b7da-63998cbf15d8 nodeName:}" failed. No retries permitted until 2026-08-04 15:10:44.895619086 +0000 UTC m=+10.249706839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bbcld" (UniqueName: "kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld") pod "network-check-target-kl72z" (UID: "86dfd1b4-818f-4c5f-b7da-63998cbf15d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:41.282784 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:41.282753 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:41.283282 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:41.282881 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:41.283362 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:41.283291 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:41.283418 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:41.283381 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:42.282168 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:42.282134 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:42.282334 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:42.282274 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:43.283500 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:43.283003 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:43.283500 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:43.283146 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:43.284272 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:43.284125 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:43.284272 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:43.284223 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:44.282766 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:44.282717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:44.282956 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.282876 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:44.727151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:44.727029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:44.727614 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.727199 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:44.727614 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.727270 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs podName:d26e3571-af26-4a3d-ab3f-0800c3b1f66b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:52.727250084 +0000 UTC m=+18.081337835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs") pod "network-metrics-daemon-w2npp" (UID: "d26e3571-af26-4a3d-ab3f-0800c3b1f66b") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:44.828271 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:44.828229 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:44.828431 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.828407 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:44.828506 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.828495 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret podName:e629fc59-eefc-49cd-9bb5-9a329b04633b nodeName:}" failed. No retries permitted until 2026-08-04 15:10:52.828464511 +0000 UTC m=+18.182552251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret") pod "global-pull-secret-syncer-6nqlc" (UID: "e629fc59-eefc-49cd-9bb5-9a329b04633b") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:44.929319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:44.928996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:44.929319 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.929200 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:10:44.929319 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.929228 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:10:44.929319 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.929240 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bbcld for pod openshift-network-diagnostics/network-check-target-kl72z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:44.929319 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:44.929321 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld podName:86dfd1b4-818f-4c5f-b7da-63998cbf15d8 nodeName:}" failed. No retries permitted until 2026-08-04 15:10:52.929302126 +0000 UTC m=+18.283389877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bbcld" (UniqueName: "kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld") pod "network-check-target-kl72z" (UID: "86dfd1b4-818f-4c5f-b7da-63998cbf15d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:45.283263 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:45.283182 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:45.283388 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:45.283266 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:45.283388 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:45.283293 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:45.283388 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:45.283373 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:45.342837 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:45.342804 2559 generic.go:356] "Generic (PLEG): container finished" podID="747b5a69ee89bac7542c3f6d437b8cbc" containerID="8f9d06867ddf6975c4baca14e1e39b81b1fd6dcf4859cefdf0a5ee13897a78c2" exitCode=0 Aug 04 15:10:45.342837 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:45.342851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" event={"ID":"747b5a69ee89bac7542c3f6d437b8cbc","Type":"ContainerDied","Data":"8f9d06867ddf6975c4baca14e1e39b81b1fd6dcf4859cefdf0a5ee13897a78c2"} Aug 04 15:10:45.362074 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:45.362022 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-35.ec2.internal" podStartSLOduration=8.362001029 podStartE2EDuration="8.362001029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:40.385892939 +0000 UTC m=+5.739980700" watchObservedRunningTime="2026-08-04 15:10:45.362001029 +0000 UTC m=+10.716088789" Aug 04 15:10:46.282232 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:46.282199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:46.282697 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:46.282341 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:47.283074 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:47.282637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:47.283074 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:47.282744 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:47.283074 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:47.282878 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:47.283074 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:47.283037 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:48.282969 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:48.282936 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:48.283141 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:48.283063 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:49.282809 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:49.282767 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:49.282993 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:49.282769 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:49.282993 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:49.282903 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:49.282993 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:49.282938 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:50.282433 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:50.282402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:50.282884 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:50.282536 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:51.282062 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:51.282028 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:51.282244 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:51.282160 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:51.282298 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:51.282245 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:51.282346 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:51.282329 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:52.282332 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:52.282245 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:52.282832 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.282394 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:52.790880 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:52.790846 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:52.791057 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.791015 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:52.791115 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.791090 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs podName:d26e3571-af26-4a3d-ab3f-0800c3b1f66b nodeName:}" failed. No retries permitted until 2026-08-04 15:11:08.791069386 +0000 UTC m=+34.145157137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs") pod "network-metrics-daemon-w2npp" (UID: "d26e3571-af26-4a3d-ab3f-0800c3b1f66b") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:10:52.892013 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:52.891979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:52.892177 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.892109 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:52.892177 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.892164 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret podName:e629fc59-eefc-49cd-9bb5-9a329b04633b nodeName:}" failed. No retries permitted until 2026-08-04 15:11:08.892151332 +0000 UTC m=+34.246239069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret") pod "global-pull-secret-syncer-6nqlc" (UID: "e629fc59-eefc-49cd-9bb5-9a329b04633b") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:10:52.993285 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.993247 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:10:52.993285 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.993278 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:10:52.993285 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.993291 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bbcld for pod openshift-network-diagnostics/network-check-target-kl72z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:52.993556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:52.993067 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:52.993556 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:52.993352 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld podName:86dfd1b4-818f-4c5f-b7da-63998cbf15d8 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:08.993334712 +0000 UTC m=+34.347422466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bbcld" (UniqueName: "kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld") pod "network-check-target-kl72z" (UID: "86dfd1b4-818f-4c5f-b7da-63998cbf15d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:10:53.285560 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:53.285529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:53.286014 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:53.285529 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:53.286014 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:53.285671 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:53.286014 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:53.285742 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:54.282795 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:54.282757 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:54.282943 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:54.282907 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:55.285019 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:55.284990 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:55.285019 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:55.285006 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:55.285546 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:55.285102 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:55.285546 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:55.285203 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:56.286268 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.285254 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:56.286268 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:56.285384 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:56.364782 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.364195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4nfzb" event={"ID":"0a1d41dd-149a-4de6-a3e7-993833e83ac3","Type":"ContainerStarted","Data":"47a3e4ea835c5ba33ac43fdd474a0277fb60401854ecb71a32f54871b8642fa2"} Aug 04 15:10:56.365595 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.365555 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerStarted","Data":"358b0615dbd653870d81b3d86beaad19d7351413f9d489cc428ccce97da5c4de"} Aug 04 15:10:56.367709 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.367689 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" event={"ID":"747b5a69ee89bac7542c3f6d437b8cbc","Type":"ContainerStarted","Data":"89198ebd66d55ec771d6134ef97886db092043d704aef08022d3c54802062647"} Aug 04 15:10:56.369125 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.369061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" event={"ID":"d1ed10dc-85d2-4dd0-9273-11e4e3d4f77f","Type":"ContainerStarted","Data":"2fe876c311b5a729b55924d12a9f9348f35b87a101197b5b38320d67c88dd3aa"} Aug 04 15:10:56.384641 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.384558 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4nfzb" podStartSLOduration=9.605589746 podStartE2EDuration="20.384545568s" totalImagesPullingTime="10.778955822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:56.384197527 +0000 UTC m=+21.738285286" watchObservedRunningTime="2026-08-04 15:10:56.384545568 +0000 UTC m=+21.738633324" Aug 04 15:10:56.404701 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.404665 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-35.ec2.internal" podStartSLOduration=19.404651654 podStartE2EDuration="19.404651654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:56.404190427 +0000 UTC m=+21.758278185" watchObservedRunningTime="2026-08-04 15:10:56.404651654 +0000 UTC m=+21.758739413" Aug 04 15:10:56.428772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:56.428736 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c7kw7" podStartSLOduration=2.7043891799999997 podStartE2EDuration="21.428722693s" totalImagesPullingTime="18.724333513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:56.428208421 +0000 UTC m=+21.782296181" watchObservedRunningTime="2026-08-04 15:10:56.428722693 +0000 UTC m=+21.782810478" Aug 04 15:10:57.285468 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.285259 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:57.285642 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.285306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:57.285642 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:57.285560 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:57.285727 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:57.285651 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:57.373403 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.373383 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:10:57.374120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.373755 2559 generic.go:356] "Generic (PLEG): container finished" podID="194c9b51-0bac-4104-8404-0437944a00f4" containerID="00d727f7090e6fcb0fa9e7099f22346030317d67d6fe7a90e999f71dcb01cf40" exitCode=1 Aug 04 15:10:57.374120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.373819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"9c0377a3fd964813dbe7edee424e1f08ce49e06fbb2b96ac30bb8c4119dab4f2"} Aug 04 15:10:57.374120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.373891 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"9bec0a28d6b739944762b7b90ae65da59e2978a5c944eca92689cbe9822adcb7"} Aug 04 15:10:57.374120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.373922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"97528c8537d32575efa4ab04fd81cabb48e52a8e972fd0a8ea30c540204970a1"} Aug 04 15:10:57.374120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.373968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"271477995f074abbdd4048cbb394a4935bcdde547f2d5c9a5a6a4007005184dc"} Aug 04 15:10:57.374120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.373997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerDied","Data":"00d727f7090e6fcb0fa9e7099f22346030317d67d6fe7a90e999f71dcb01cf40"} Aug 04 15:10:57.374120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.374033 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"0f0ee268a7ee9f1d63d94a90ac22baa5987ec2879c9e48b8d72887f280d95b9e"} Aug 04 15:10:57.376410 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.376388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wzhmd" event={"ID":"ab0ee247-651d-4fd2-a2ad-af1ede75d32e","Type":"ContainerStarted","Data":"498f97e2e86fc6bbe9f21dc0141770e9df12b973d9452044415e378ee3d891c0"} Aug 04 15:10:57.377770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.377752 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f34acf7-da6c-4fb9-aad4-dff78830a97e" containerID="358b0615dbd653870d81b3d86beaad19d7351413f9d489cc428ccce97da5c4de" exitCode=0 Aug 04 15:10:57.377873 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.377810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerDied","Data":"358b0615dbd653870d81b3d86beaad19d7351413f9d489cc428ccce97da5c4de"} Aug 04 15:10:57.379206 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.379186 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lk76r" event={"ID":"fa1d177c-d3ac-425d-a836-58dc1dc7b425","Type":"ContainerStarted","Data":"11a95f1544f8e4b98ce134a7f63958ff83047b62d2f65f8d102291d705f270c9"} Aug 04 15:10:57.380444 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.380424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j8h9m" event={"ID":"7632fc34-ddbf-4d83-8048-0a9d86cdf1a0","Type":"ContainerStarted","Data":"7eea8b87d061d8cea4d30055a4d108ede0bfb97e8b9f88f07a6aeafb1da95cd1"} Aug 04 15:10:57.381827 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.381804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" event={"ID":"4506fb2a-2d1b-4df3-b915-682295356ecd","Type":"ContainerStarted","Data":"2cfd58b82aef0c5a59ee3761942c942fee6328a85d6f6dbc7e35fcf7b39171c1"} Aug 04 15:10:57.432790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.432725 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wzhmd" podStartSLOduration=3.664819679 podStartE2EDuration="22.432712052s" totalImagesPullingTime="18.767892373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:57.412463908 +0000 UTC m=+22.766551670" watchObservedRunningTime="2026-08-04 15:10:57.432712052 +0000 UTC m=+22.786799810" Aug 04 15:10:57.461159 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.461123 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lk76r" podStartSLOduration=3.7117229309999997 podStartE2EDuration="22.461111351s" totalImagesPullingTime="18.74938842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:57.432831537 +0000 UTC m=+22.786919274" watchObservedRunningTime="2026-08-04 15:10:57.461111351 +0000 UTC m=+22.815199109" Aug 04 15:10:57.491679 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.491646 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j8h9m" podStartSLOduration=3.730806988 podStartE2EDuration="22.491638624s" totalImagesPullingTime="18.760831636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:57.491389794 +0000 UTC m=+22.845477553" watchObservedRunningTime="2026-08-04 15:10:57.491638624 +0000 UTC m=+22.845726383" Aug 04 15:10:57.710544 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:57.710523 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Aug 04 15:10:58.108024 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.107877 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T15:10:57.710540761Z","UUID":"88a2871b-2a80-4dbd-8508-6f42846af2fe","Handler":null,"Name":"","Endpoint":""} Aug 04 15:10:58.109488 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.109466 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Aug 04 15:10:58.109620 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.109504 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Aug 04 15:10:58.282675 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.282638 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:10:58.282846 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:58.282765 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:10:58.386285 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.386205 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" event={"ID":"4506fb2a-2d1b-4df3-b915-682295356ecd","Type":"ContainerStarted","Data":"d980169a399c70f65b393ce63c77cfd45f91d94a81f34855e3ba7eae38797bad"} Aug 04 15:10:58.388280 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.388168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-btv2r" event={"ID":"39ce0ab6-9006-4378-b717-9cae3c2e27ee","Type":"ContainerStarted","Data":"2d41d485d41e7e936986ff0619ef32932f3372d75fef8c5ef55dd9cbebafac4f"} Aug 04 15:10:58.419335 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.419277 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-btv2r" podStartSLOduration=3.723875106 podStartE2EDuration="22.419256583s" totalImagesPullingTime="18.695381477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:58.418676726 +0000 UTC m=+23.772764488" watchObservedRunningTime="2026-08-04 15:10:58.419256583 +0000 UTC m=+23.773344343" Aug 04 15:10:58.457927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:58.457890 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:59.050375 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.050342 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:59.051048 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.051026 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:59.282422 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.282399 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:10:59.282519 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:59.282499 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:10:59.282586 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.282550 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:10:59.282662 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:10:59.282645 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:10:59.392804 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.392611 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" event={"ID":"4506fb2a-2d1b-4df3-b915-682295356ecd","Type":"ContainerStarted","Data":"2bd1730abfd4c6fd17e9b05c2cf1fdccbcf88f3d708d8de0f9f74f2fb51645db"} Aug 04 15:10:59.395646 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.395626 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:10:59.396010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.395992 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"70a9e34bd05478ecc4d6df323f31774a3075bfc5d62d1a4c382f8898d9a84a75"} Aug 04 15:10:59.396711 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.396693 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4nfzb" Aug 04 15:10:59.433953 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:10:59.433913 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-krsk7" podStartSLOduration=2.092246785 podStartE2EDuration="23.433898667s" totalImagesPullingTime="21.341651882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:10:59.433782697 +0000 UTC m=+24.787870456" watchObservedRunningTime="2026-08-04 15:10:59.433898667 +0000 UTC m=+24.787986427" Aug 04 15:11:00.282356 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:00.282325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:00.282558 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:00.282436 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:11:01.282203 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:01.282169 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:01.282696 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:01.282169 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:01.282696 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:01.282285 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:11:01.282696 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:01.282339 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:11:02.282562 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:02.282311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:02.283233 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:02.282699 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:11:02.406567 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:02.406536 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:11:02.406986 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:02.406965 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"de0b4a9bd507e21b5a0adb3b60a74ab0ab4d16ce025c3bbd74e23ba1dc0fb41b"} Aug 04 15:11:02.407270 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:02.407244 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:11:02.407394 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:02.407319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:11:02.407449 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:02.407419 2559 scope.go:117] "RemoveContainer" containerID="00d727f7090e6fcb0fa9e7099f22346030317d67d6fe7a90e999f71dcb01cf40" Aug 04 15:11:02.425726 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:02.425704 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:11:03.282805 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.282773 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:03.283217 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:03.282906 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:11:03.283217 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.282937 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:03.283217 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:03.283052 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:11:03.412956 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.412936 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:11:03.413366 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.413346 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" event={"ID":"194c9b51-0bac-4104-8404-0437944a00f4","Type":"ContainerStarted","Data":"a830fa69f37e927a882eae792fa6a818fd974904060884d3aa13e2e5c53ddc6e"} Aug 04 15:11:03.413622 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.413606 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:11:03.430122 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.430055 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:11:03.506797 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.506749 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" podStartSLOduration=8.323531599 podStartE2EDuration="27.506732348s" totalImagesPullingTime="19.183200749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:03.459223776 +0000 UTC m=+28.813311538" watchObservedRunningTime="2026-08-04 15:11:03.506732348 +0000 UTC m=+28.860820107" Aug 04 15:11:03.882651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.882621 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nqlc"] Aug 04 15:11:03.882811 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.882739 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:03.882858 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:03.882836 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:11:03.886661 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.886549 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w2npp"] Aug 04 15:11:03.886782 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.886684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:03.886835 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:03.886796 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:11:03.902761 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.902731 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kl72z"] Aug 04 15:11:03.902891 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:03.902851 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:03.902968 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:03.902946 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:11:04.417201 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:04.417170 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f34acf7-da6c-4fb9-aad4-dff78830a97e" containerID="6c574a8845117465cf24f39ca5cc21ea0ac1890e100fab4e8f4f403948e48222" exitCode=0 Aug 04 15:11:04.417633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:04.417256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerDied","Data":"6c574a8845117465cf24f39ca5cc21ea0ac1890e100fab4e8f4f403948e48222"} Aug 04 15:11:05.283379 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:05.283349 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:05.283539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:05.283432 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:05.283611 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:05.283432 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:11:05.283611 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:05.283547 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:11:06.282422 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:06.282254 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:06.282927 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:06.282490 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:11:06.423445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:06.423411 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f34acf7-da6c-4fb9-aad4-dff78830a97e" containerID="fad61da3ad189d34b117d44de227e81bc10685f14c83f251c22937945fe46bcc" exitCode=0 Aug 04 15:11:06.423600 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:06.423470 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerDied","Data":"fad61da3ad189d34b117d44de227e81bc10685f14c83f251c22937945fe46bcc"} Aug 04 15:11:07.282821 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:07.282777 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:07.282821 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:07.282814 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:07.283332 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:07.282918 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:11:07.283332 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:07.283028 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:11:08.282518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:08.282480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:08.282690 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:08.282624 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kl72z" podUID="86dfd1b4-818f-4c5f-b7da-63998cbf15d8" Aug 04 15:11:08.430402 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:08.430368 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f34acf7-da6c-4fb9-aad4-dff78830a97e" containerID="ff04cd8dbbdd944bcf258331a2ef08bcafcbc6ee894b874ffe8c829392be1e96" exitCode=0 Aug 04 15:11:08.430896 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:08.430407 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerDied","Data":"ff04cd8dbbdd944bcf258331a2ef08bcafcbc6ee894b874ffe8c829392be1e96"} Aug 04 15:11:08.812819 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:08.812790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:08.813014 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:08.812950 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:11:08.813014 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:08.813010 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs podName:d26e3571-af26-4a3d-ab3f-0800c3b1f66b nodeName:}" failed. No retries permitted until 2026-08-04 15:11:40.812995923 +0000 UTC m=+66.167083660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs") pod "network-metrics-daemon-w2npp" (UID: "d26e3571-af26-4a3d-ab3f-0800c3b1f66b") : object "openshift-multus"/"metrics-daemon-secret" not registered Aug 04 15:11:08.913855 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:08.913816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:08.914030 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:08.913963 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Aug 04 15:11:08.914030 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:08.914021 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret podName:e629fc59-eefc-49cd-9bb5-9a329b04633b nodeName:}" failed. No retries permitted until 2026-08-04 15:11:40.914006021 +0000 UTC m=+66.268093762 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret") pod "global-pull-secret-syncer-6nqlc" (UID: "e629fc59-eefc-49cd-9bb5-9a329b04633b") : object "kube-system"/"original-pull-secret" not registered Aug 04 15:11:09.014462 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.014420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:09.014657 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:09.014620 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Aug 04 15:11:09.014657 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:09.014647 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Aug 04 15:11:09.014657 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:09.014660 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bbcld for pod openshift-network-diagnostics/network-check-target-kl72z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:11:09.014769 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:09.014726 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld podName:86dfd1b4-818f-4c5f-b7da-63998cbf15d8 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:41.014708032 +0000 UTC m=+66.368795771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bbcld" (UniqueName: "kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld") pod "network-check-target-kl72z" (UID: "86dfd1b4-818f-4c5f-b7da-63998cbf15d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Aug 04 15:11:09.282865 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.282830 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:09.282865 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.282866 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:09.283115 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:09.282975 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w2npp" podUID="d26e3571-af26-4a3d-ab3f-0800c3b1f66b" Aug 04 15:11:09.283115 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:09.283079 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6nqlc" podUID="e629fc59-eefc-49cd-9bb5-9a329b04633b" Aug 04 15:11:09.930572 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.930542 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-35.ec2.internal" event="NodeReady" Aug 04 15:11:09.931052 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.930713 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Aug 04 15:11:09.976365 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.976221 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bb959c8b7-dv5dw"] Aug 04 15:11:09.990025 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.990001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:09.992968 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.992945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Aug 04 15:11:09.993084 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.992951 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Aug 04 15:11:09.994047 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.993762 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cqks7\"" Aug 04 15:11:09.994047 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.993931 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Aug 04 15:11:09.999978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:09.999956 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bb959c8b7-dv5dw"] Aug 04 15:11:10.000114 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.000098 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Aug 04 15:11:10.002679 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.002660 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mlgv6"] Aug 04 15:11:10.018661 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.017844 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2bng9"] Aug 04 15:11:10.018772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.018706 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.022324 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.022117 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kdxng\"" Aug 04 15:11:10.022324 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.022125 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Aug 04 15:11:10.022758 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.022596 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Aug 04 15:11:10.022758 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.022658 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Aug 04 15:11:10.032197 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.032176 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mlgv6"] Aug 04 15:11:10.032288 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.032213 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bng9"] Aug 04 15:11:10.032288 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.032277 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.038231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.038206 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Aug 04 15:11:10.038811 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.038360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xckv6\"" Aug 04 15:11:10.039183 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.039162 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Aug 04 15:11:10.125382 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-ca-trust-extracted\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.125382 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125343 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2vhj\" (UniqueName: \"kubernetes.io/projected/edc4b4c4-6033-4d2d-87ff-c1be81027a01-kube-api-access-q2vhj\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.125382 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125371 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-certificates\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.125671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-bound-sa-token\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.125671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc4b4c4-6033-4d2d-87ff-c1be81027a01-config-volume\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.125671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc4b4c4-6033-4d2d-87ff-c1be81027a01-tmp-dir\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.125671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125622 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.125671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125655 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-image-registry-private-configuration\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.125922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125690 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv9zv\" (UniqueName: \"kubernetes.io/projected/c389923e-0891-45d8-b662-1700d98d597b-kube-api-access-hv9zv\") pod \"ingress-canary-mlgv6\" (UID: \"c389923e-0891-45d8-b662-1700d98d597b\") " pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.125922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-trusted-ca\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.125922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125782 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-installation-pull-secrets\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.125922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2czll\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-kube-api-access-2czll\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.125922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c389923e-0891-45d8-b662-1700d98d597b-cert\") pod \"ingress-canary-mlgv6\" (UID: \"c389923e-0891-45d8-b662-1700d98d597b\") " pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.125922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.125895 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.226822 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.226783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-ca-trust-extracted\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.226822 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.226829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2vhj\" (UniqueName: \"kubernetes.io/projected/edc4b4c4-6033-4d2d-87ff-c1be81027a01-kube-api-access-q2vhj\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.227049 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.226856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-certificates\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227049 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.226876 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-bound-sa-token\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc4b4c4-6033-4d2d-87ff-c1be81027a01-config-volume\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.227151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc4b4c4-6033-4d2d-87ff-c1be81027a01-tmp-dir\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.227151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.227302 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-image-registry-private-configuration\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227302 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227205 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-ca-trust-extracted\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227302 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv9zv\" (UniqueName: \"kubernetes.io/projected/c389923e-0891-45d8-b662-1700d98d597b-kube-api-access-hv9zv\") pod \"ingress-canary-mlgv6\" (UID: \"c389923e-0891-45d8-b662-1700d98d597b\") " pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.227302 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.227264 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:11:10.227302 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-trusted-ca\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-installation-pull-secrets\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2czll\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-kube-api-access-2czll\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c389923e-0891-45d8-b662-1700d98d597b-cert\") pod \"ingress-canary-mlgv6\" (UID: \"c389923e-0891-45d8-b662-1700d98d597b\") " pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.227545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227389 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edc4b4c4-6033-4d2d-87ff-c1be81027a01-tmp-dir\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.227545 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.227457 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls podName:edc4b4c4-6033-4d2d-87ff-c1be81027a01 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:10.727442158 +0000 UTC m=+36.081529895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls") pod "dns-default-2bng9" (UID: "edc4b4c4-6033-4d2d-87ff-c1be81027a01") : secret "dns-default-metrics-tls" not found Aug 04 15:11:10.227545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227476 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-certificates\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.227915 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.227667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edc4b4c4-6033-4d2d-87ff-c1be81027a01-config-volume\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.227915 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.227822 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:11:10.227915 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.227834 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb959c8b7-dv5dw: secret "image-registry-tls" not found Aug 04 15:11:10.227915 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.227884 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls podName:3c793f63-8a0d-40e5-8f6a-61b6ce8a7931 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:10.72786701 +0000 UTC m=+36.081954750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls") pod "image-registry-7bb959c8b7-dv5dw" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931") : secret "image-registry-tls" not found Aug 04 15:11:10.228240 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.228219 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-trusted-ca\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.231657 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.231629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-image-registry-private-configuration\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.231772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.231670 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-installation-pull-secrets\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.231772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.231698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c389923e-0891-45d8-b662-1700d98d597b-cert\") pod \"ingress-canary-mlgv6\" (UID: \"c389923e-0891-45d8-b662-1700d98d597b\") " pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.240090 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.240062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv9zv\" (UniqueName: \"kubernetes.io/projected/c389923e-0891-45d8-b662-1700d98d597b-kube-api-access-hv9zv\") pod \"ingress-canary-mlgv6\" (UID: \"c389923e-0891-45d8-b662-1700d98d597b\") " pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.240489 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.240443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2czll\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-kube-api-access-2czll\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.240624 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.240501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2vhj\" (UniqueName: \"kubernetes.io/projected/edc4b4c4-6033-4d2d-87ff-c1be81027a01-kube-api-access-q2vhj\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.240624 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.240616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-bound-sa-token\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.282667 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.282627 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:10.285888 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.285864 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Aug 04 15:11:10.286139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.286118 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Aug 04 15:11:10.286406 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.286379 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l7r4m\"" Aug 04 15:11:10.328663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.328637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mlgv6" Aug 04 15:11:10.538384 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.538186 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mlgv6"] Aug 04 15:11:10.541560 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:10.541528 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc389923e_0891_45d8_b662_1700d98d597b.slice/crio-2baec137193515a39c81531a307e1cf41ccede032d78fa11514216555e1c9da8 WatchSource:0}: Error finding container 2baec137193515a39c81531a307e1cf41ccede032d78fa11514216555e1c9da8: Status 404 returned error can't find the container with id 2baec137193515a39c81531a307e1cf41ccede032d78fa11514216555e1c9da8 Aug 04 15:11:10.730647 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.730524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:10.730807 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:10.730652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:10.730807 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.730707 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:11:10.730807 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.730729 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb959c8b7-dv5dw: secret "image-registry-tls" not found Aug 04 15:11:10.730807 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.730773 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:11:10.730807 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.730796 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls podName:3c793f63-8a0d-40e5-8f6a-61b6ce8a7931 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:11.730775143 +0000 UTC m=+37.084862897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls") pod "image-registry-7bb959c8b7-dv5dw" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931") : secret "image-registry-tls" not found Aug 04 15:11:10.731001 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:10.730823 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls podName:edc4b4c4-6033-4d2d-87ff-c1be81027a01 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:11.730807745 +0000 UTC m=+37.084895503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls") pod "dns-default-2bng9" (UID: "edc4b4c4-6033-4d2d-87ff-c1be81027a01") : secret "dns-default-metrics-tls" not found Aug 04 15:11:11.282678 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.282646 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:11.282678 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.282673 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:11.286246 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.286228 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 15:11:11.286472 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.286460 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 15:11:11.287053 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.287038 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8gfq\"" Aug 04 15:11:11.438435 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.438402 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mlgv6" event={"ID":"c389923e-0891-45d8-b662-1700d98d597b","Type":"ContainerStarted","Data":"2baec137193515a39c81531a307e1cf41ccede032d78fa11514216555e1c9da8"} Aug 04 15:11:11.738784 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.738745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:11.738951 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.738849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:11.738951 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:11.738891 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:11:11.738951 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:11.738916 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb959c8b7-dv5dw: secret "image-registry-tls" not found Aug 04 15:11:11.738951 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:11.738950 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:11:11.739144 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:11.738982 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls podName:3c793f63-8a0d-40e5-8f6a-61b6ce8a7931 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:13.738963185 +0000 UTC m=+39.093050937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls") pod "image-registry-7bb959c8b7-dv5dw" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931") : secret "image-registry-tls" not found Aug 04 15:11:11.739144 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:11.738996 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls podName:edc4b4c4-6033-4d2d-87ff-c1be81027a01 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:13.738990881 +0000 UTC m=+39.093078618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls") pod "dns-default-2bng9" (UID: "edc4b4c4-6033-4d2d-87ff-c1be81027a01") : secret "dns-default-metrics-tls" not found Aug 04 15:11:11.829633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:11.829515 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:12.061555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:12.061473 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:12.176381 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:12.176343 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:12.289787 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:12.289749 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:12.691235 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:12.691200 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:12.774188 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:12.774156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:13.096465 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:13.096435 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:13.184293 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:13.184253 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:13.755679 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:13.755642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:13.756140 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:13.755721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:13.756140 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:13.755778 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:11:13.756140 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:13.755835 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:11:13.756140 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:13.755850 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb959c8b7-dv5dw: secret "image-registry-tls" not found Aug 04 15:11:13.756140 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:13.755861 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls podName:edc4b4c4-6033-4d2d-87ff-c1be81027a01 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:17.755838237 +0000 UTC m=+43.109925981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls") pod "dns-default-2bng9" (UID: "edc4b4c4-6033-4d2d-87ff-c1be81027a01") : secret "dns-default-metrics-tls" not found Aug 04 15:11:13.756140 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:13.755901 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls podName:3c793f63-8a0d-40e5-8f6a-61b6ce8a7931 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:17.755885471 +0000 UTC m=+43.109973214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls") pod "image-registry-7bb959c8b7-dv5dw" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931") : secret "image-registry-tls" not found Aug 04 15:11:14.258214 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:14.258173 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:14.354999 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:14.354968 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:14.450872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:14.450806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerStarted","Data":"58375544a5591d4710641d330a8fb65ffc69e204a25618627882431347cef6da"} Aug 04 15:11:14.454888 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:14.454838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mlgv6" event={"ID":"c389923e-0891-45d8-b662-1700d98d597b","Type":"ContainerStarted","Data":"6dd6d8f5623209c4e7bed61a86d88c8bf50680646b7e21060a609b9ac2f424eb"} Aug 04 15:11:14.498193 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:14.498143 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mlgv6" podStartSLOduration=1.7752848430000001 podStartE2EDuration="5.498127406s" totalImagesPullingTime="3.722842563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:14.497563595 +0000 UTC m=+39.851651356" watchObservedRunningTime="2026-08-04 15:11:14.498127406 +0000 UTC m=+39.852215164" Aug 04 15:11:15.459585 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:15.459553 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f34acf7-da6c-4fb9-aad4-dff78830a97e" containerID="58375544a5591d4710641d330a8fb65ffc69e204a25618627882431347cef6da" exitCode=0 Aug 04 15:11:15.459940 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:15.459607 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerDied","Data":"58375544a5591d4710641d330a8fb65ffc69e204a25618627882431347cef6da"} Aug 04 15:11:16.467280 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:16.467247 2559 generic.go:356] "Generic (PLEG): container finished" podID="1f34acf7-da6c-4fb9-aad4-dff78830a97e" containerID="3a2ea5aa9c20769caf28081abf63e023af8d7d1c04f2b57e92fbe5b496ac3c58" exitCode=0 Aug 04 15:11:16.467713 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:16.467305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerDied","Data":"3a2ea5aa9c20769caf28081abf63e023af8d7d1c04f2b57e92fbe5b496ac3c58"} Aug 04 15:11:17.472745 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.472701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" event={"ID":"1f34acf7-da6c-4fb9-aad4-dff78830a97e","Type":"ContainerStarted","Data":"2d48bb81bfcaa2215c593c21d6c77e37acf5a37132f4804711f5982133c47953"} Aug 04 15:11:17.509210 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.509164 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vk2fg" podStartSLOduration=9.562985459 podStartE2EDuration="42.509147914s" totalImagesPullingTime="32.946162455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:17.507154656 +0000 UTC m=+42.861242414" watchObservedRunningTime="2026-08-04 15:11:17.509147914 +0000 UTC m=+42.863235673" Aug 04 15:11:17.653397 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.653360 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:17.753397 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.753318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:17.786544 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.786513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:17.786704 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.786685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:17.786757 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:17.786687 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:11:17.786757 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:17.786745 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb959c8b7-dv5dw: secret "image-registry-tls" not found Aug 04 15:11:17.786757 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:17.786730 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:11:17.786849 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:17.786795 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls podName:3c793f63-8a0d-40e5-8f6a-61b6ce8a7931 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:25.78678063 +0000 UTC m=+51.140868368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls") pod "image-registry-7bb959c8b7-dv5dw" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931") : secret "image-registry-tls" not found Aug 04 15:11:17.786849 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:17.786808 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls podName:edc4b4c4-6033-4d2d-87ff-c1be81027a01 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:25.786802793 +0000 UTC m=+51.140890529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls") pod "dns-default-2bng9" (UID: "edc4b4c4-6033-4d2d-87ff-c1be81027a01") : secret "dns-default-metrics-tls" not found Aug 04 15:11:17.895571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.895540 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-z5kg4"] Aug 04 15:11:17.924674 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.924647 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-z5kg4"] Aug 04 15:11:17.924817 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.924747 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:17.928122 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.928101 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j55g8\"" Aug 04 15:11:17.928473 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.928454 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Aug 04 15:11:17.928537 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.928495 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Aug 04 15:11:17.928627 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.928610 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Aug 04 15:11:17.928787 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.928770 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Aug 04 15:11:17.936792 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.935236 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Aug 04 15:11:17.992747 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:17.992715 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z"] Aug 04 15:11:18.011210 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.011140 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-94v75"] Aug 04 15:11:18.011482 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.011290 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.014473 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.014318 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dblpb\"" Aug 04 15:11:18.014473 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.014318 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Aug 04 15:11:18.014662 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.014624 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Aug 04 15:11:18.015543 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.015527 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Aug 04 15:11:18.015650 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.015591 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Aug 04 15:11:18.024984 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.024965 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z"] Aug 04 15:11:18.025072 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.024998 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-94v75"] Aug 04 15:11:18.025116 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.025077 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" Aug 04 15:11:18.028076 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.028060 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cltzb\"" Aug 04 15:11:18.089691 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.089659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-trusted-ca-bundle\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.089691 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.089690 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6blp\" (UniqueName: \"kubernetes.io/projected/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-kube-api-access-d6blp\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.089927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.089731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-serving-cert\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.089927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.089834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-tmp\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.089927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.089869 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-service-ca-bundle\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.089927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.089922 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-snapshots\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.105128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.105100 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl"] Aug 04 15:11:18.127007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.126980 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:18.129518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.129491 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl"] Aug 04 15:11:18.131554 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.131529 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:11:18.131554 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.131549 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Aug 04 15:11:18.131709 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.131571 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Aug 04 15:11:18.131709 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.131543 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5wb6s\"" Aug 04 15:11:18.140604 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.140562 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59fcd9fd6d-l6gbg"] Aug 04 15:11:18.161073 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.161042 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b"] Aug 04 15:11:18.161190 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.161167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.164752 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.164399 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Aug 04 15:11:18.164752 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.164446 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Aug 04 15:11:18.164752 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.164405 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Aug 04 15:11:18.165086 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.164807 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Aug 04 15:11:18.165086 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.165030 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Aug 04 15:11:18.165326 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.165304 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Aug 04 15:11:18.166390 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.166361 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dndng\"" Aug 04 15:11:18.177131 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.177113 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp"] Aug 04 15:11:18.177245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.177228 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" Aug 04 15:11:18.181157 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.181137 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Aug 04 15:11:18.181246 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.181139 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-njzp8\"" Aug 04 15:11:18.181246 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.181139 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:11:18.190712 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.190690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-service-ca-bundle\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.190793 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.190736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.190862 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.190844 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-snapshots\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.190925 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.190908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-trusted-ca-bundle\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.191241 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6blp\" (UniqueName: \"kubernetes.io/projected/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-kube-api-access-d6blp\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.191418 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-serving-cert\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.191505 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5rh\" (UniqueName: \"kubernetes.io/projected/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-kube-api-access-qq5rh\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.191505 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191498 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.191638 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjgq\" (UniqueName: \"kubernetes.io/projected/6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e-kube-api-access-sqjgq\") pod \"network-check-source-65bdbbff45-94v75\" (UID: \"6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" Aug 04 15:11:18.191638 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191541 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-service-ca-bundle\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.191795 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191769 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-snapshots\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.191975 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.191947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-tmp\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.192076 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.192049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-tmp\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.192846 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.192703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-trusted-ca-bundle\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.195967 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.195949 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-serving-cert\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.199379 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.199357 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp"] Aug 04 15:11:18.199465 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.199402 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b"] Aug 04 15:11:18.199465 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.199420 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59fcd9fd6d-l6gbg"] Aug 04 15:11:18.199567 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.199511 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.204220 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.204185 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Aug 04 15:11:18.204318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.204229 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Aug 04 15:11:18.206350 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.206330 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8lhtr\"" Aug 04 15:11:18.206442 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.206389 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Aug 04 15:11:18.206442 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.206390 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:11:18.212938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.212917 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6blp\" (UniqueName: \"kubernetes.io/projected/e8b23c6f-dabe-4c2d-9993-e7682fdb4c26-kube-api-access-d6blp\") pod \"insights-operator-77696685b-z5kg4\" (UID: \"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26\") " pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.221533 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.221514 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz"] Aug 04 15:11:18.233862 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.233837 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-z5kg4" Aug 04 15:11:18.253653 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.253629 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-9j44d"] Aug 04 15:11:18.253781 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.253754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.256271 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.256239 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Aug 04 15:11:18.256404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.256316 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:11:18.256404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.256327 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Aug 04 15:11:18.256404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.256322 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-qwrm6\"" Aug 04 15:11:18.256404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.256380 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Aug 04 15:11:18.270978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.270957 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz"] Aug 04 15:11:18.271073 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.270984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-9j44d"] Aug 04 15:11:18.271108 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.271069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.273975 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.273956 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Aug 04 15:11:18.274351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.274331 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Aug 04 15:11:18.274712 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.274693 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g47dx\"" Aug 04 15:11:18.274796 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.274782 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Aug 04 15:11:18.275558 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.275545 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Aug 04 15:11:18.279790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.279763 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Aug 04 15:11:18.292600 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292563 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfwr2\" (UniqueName: \"kubernetes.io/projected/dc1eb654-576c-44bd-ab10-f7a2ed3d75e3-kube-api-access-zfwr2\") pod \"volume-data-source-validator-5b8776669-k789b\" (UID: \"dc1eb654-576c-44bd-ab10-f7a2ed3d75e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" Aug 04 15:11:18.292723 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.292723 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6fsw\" (UniqueName: \"kubernetes.io/projected/2af465a1-b59e-4dc0-99c2-828508ac8a4c-kube-api-access-t6fsw\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:18.292723 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292721 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:18.292878 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292745 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-default-certificate\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.292878 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292760 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk9k4\" (UniqueName: \"kubernetes.io/projected/c330d986-f440-4bb4-b152-bdb21af2925e-kube-api-access-qk9k4\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.292878 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.292878 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.292814 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:18.293060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjgq\" (UniqueName: \"kubernetes.io/projected/6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e-kube-api-access-sqjgq\") pod \"network-check-source-65bdbbff45-94v75\" (UID: \"6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" Aug 04 15:11:18.293060 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.292878 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls podName:f2fb7ca7-9668-4869-a3df-5022e5c62cf1 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:18.792860378 +0000 UTC m=+44.146948119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-sxq7z" (UID: "f2fb7ca7-9668-4869-a3df-5022e5c62cf1") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:18.293060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.292990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5rh\" (UniqueName: \"kubernetes.io/projected/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-kube-api-access-qq5rh\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.293060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.293021 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-stats-auth\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.293060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.293046 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.293255 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.293096 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.293425 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.293407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.312948 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.312922 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5rh\" (UniqueName: \"kubernetes.io/projected/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-kube-api-access-qq5rh\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.315462 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.315442 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjgq\" (UniqueName: \"kubernetes.io/projected/6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e-kube-api-access-sqjgq\") pod \"network-check-source-65bdbbff45-94v75\" (UID: \"6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" Aug 04 15:11:18.332375 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.332351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" Aug 04 15:11:18.384626 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.384591 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:18.393674 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.393597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6fsw\" (UniqueName: \"kubernetes.io/projected/2af465a1-b59e-4dc0-99c2-828508ac8a4c-kube-api-access-t6fsw\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:18.393674 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.393636 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/814798a3-c8db-4727-920f-dfca65127d18-serving-cert\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.393843 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.393694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:18.393843 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.393735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/814798a3-c8db-4727-920f-dfca65127d18-config\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.393843 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.393767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvq6l\" (UniqueName: \"kubernetes.io/projected/30c90e46-5199-4d39-8858-910139a1cbcf-kube-api-access-pvq6l\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.393987 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.393894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30c90e46-5199-4d39-8858-910139a1cbcf-config\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.393987 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.393977 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 15:11:18.394073 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.394045 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls podName:2af465a1-b59e-4dc0-99c2-828508ac8a4c nodeName:}" failed. No retries permitted until 2026-08-04 15:11:18.894023376 +0000 UTC m=+44.248111113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-wj5cl" (UID: "2af465a1-b59e-4dc0-99c2-828508ac8a4c") : secret "samples-operator-tls" not found Aug 04 15:11:18.394073 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-default-certificate\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.394146 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk9k4\" (UniqueName: \"kubernetes.io/projected/c330d986-f440-4bb4-b152-bdb21af2925e-kube-api-access-qk9k4\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.394146 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/814798a3-c8db-4727-920f-dfca65127d18-trusted-ca\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.394208 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48fk\" (UniqueName: \"kubernetes.io/projected/814798a3-c8db-4727-920f-dfca65127d18-kube-api-access-p48fk\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.394208 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30c90e46-5199-4d39-8858-910139a1cbcf-serving-cert\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.394265 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394239 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f257daa1-126f-4549-a16f-4a5967506834-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.394300 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-stats-auth\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.394300 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.394380 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.394417 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfwr2\" (UniqueName: \"kubernetes.io/projected/dc1eb654-576c-44bd-ab10-f7a2ed3d75e3-kube-api-access-zfwr2\") pod \"volume-data-source-validator-5b8776669-k789b\" (UID: \"dc1eb654-576c-44bd-ab10-f7a2ed3d75e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" Aug 04 15:11:18.394449 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394415 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l5hq\" (UniqueName: \"kubernetes.io/projected/f257daa1-126f-4549-a16f-4a5967506834-kube-api-access-2l5hq\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.394485 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.394475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f257daa1-126f-4549-a16f-4a5967506834-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.394763 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.394681 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:11:18.394763 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.394697 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:18.894680786 +0000 UTC m=+44.248768535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : configmap references non-existent config key: service-ca.crt Aug 04 15:11:18.394763 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.394725 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:18.894712693 +0000 UTC m=+44.248800448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : secret "router-metrics-certs-default" not found Aug 04 15:11:18.396976 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.396950 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-default-certificate\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.398007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.397988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-stats-auth\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.404023 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.404000 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk9k4\" (UniqueName: \"kubernetes.io/projected/c330d986-f440-4bb4-b152-bdb21af2925e-kube-api-access-qk9k4\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.404528 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.404504 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6fsw\" (UniqueName: \"kubernetes.io/projected/2af465a1-b59e-4dc0-99c2-828508ac8a4c-kube-api-access-t6fsw\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:18.409164 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.409138 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-z5kg4"] Aug 04 15:11:18.410889 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.410870 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfwr2\" (UniqueName: \"kubernetes.io/projected/dc1eb654-576c-44bd-ab10-f7a2ed3d75e3-kube-api-access-zfwr2\") pod \"volume-data-source-validator-5b8776669-k789b\" (UID: \"dc1eb654-576c-44bd-ab10-f7a2ed3d75e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" Aug 04 15:11:18.412364 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:18.412339 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b23c6f_dabe_4c2d_9993_e7682fdb4c26.slice/crio-626920f4866ff130d8f0a24fbc01eccc717118826cc0635e0af2d62e26da621d WatchSource:0}: Error finding container 626920f4866ff130d8f0a24fbc01eccc717118826cc0635e0af2d62e26da621d: Status 404 returned error can't find the container with id 626920f4866ff130d8f0a24fbc01eccc717118826cc0635e0af2d62e26da621d Aug 04 15:11:18.468654 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.468625 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-94v75"] Aug 04 15:11:18.471514 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:18.471488 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d83eac4_1c69_43f5_9f49_7e4b6cf1a92e.slice/crio-ba33720c5d110341a2cf5d87c9382c9f60c3d44c336ecc3de8e72932a1a68440 WatchSource:0}: Error finding container ba33720c5d110341a2cf5d87c9382c9f60c3d44c336ecc3de8e72932a1a68440: Status 404 returned error can't find the container with id ba33720c5d110341a2cf5d87c9382c9f60c3d44c336ecc3de8e72932a1a68440 Aug 04 15:11:18.475720 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.475693 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" event={"ID":"6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e","Type":"ContainerStarted","Data":"ba33720c5d110341a2cf5d87c9382c9f60c3d44c336ecc3de8e72932a1a68440"} Aug 04 15:11:18.476245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.476216 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:18.476919 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.476877 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-z5kg4" event={"ID":"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26","Type":"ContainerStarted","Data":"626920f4866ff130d8f0a24fbc01eccc717118826cc0635e0af2d62e26da621d"} Aug 04 15:11:18.484823 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.484806 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" Aug 04 15:11:18.495294 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f257daa1-126f-4549-a16f-4a5967506834-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.495378 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495335 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l5hq\" (UniqueName: \"kubernetes.io/projected/f257daa1-126f-4549-a16f-4a5967506834-kube-api-access-2l5hq\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.495534 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f257daa1-126f-4549-a16f-4a5967506834-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.495626 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495558 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/814798a3-c8db-4727-920f-dfca65127d18-serving-cert\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.495718 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/814798a3-c8db-4727-920f-dfca65127d18-config\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.495771 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvq6l\" (UniqueName: \"kubernetes.io/projected/30c90e46-5199-4d39-8858-910139a1cbcf-kube-api-access-pvq6l\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.495771 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30c90e46-5199-4d39-8858-910139a1cbcf-config\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.495882 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495773 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/814798a3-c8db-4727-920f-dfca65127d18-trusted-ca\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.495882 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48fk\" (UniqueName: \"kubernetes.io/projected/814798a3-c8db-4727-920f-dfca65127d18-kube-api-access-p48fk\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.495882 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495838 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30c90e46-5199-4d39-8858-910139a1cbcf-serving-cert\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.496021 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.495947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f257daa1-126f-4549-a16f-4a5967506834-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.496384 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.496361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30c90e46-5199-4d39-8858-910139a1cbcf-config\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.496528 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.496464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/814798a3-c8db-4727-920f-dfca65127d18-config\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.496664 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.496644 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/814798a3-c8db-4727-920f-dfca65127d18-trusted-ca\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.497821 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.497798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30c90e46-5199-4d39-8858-910139a1cbcf-serving-cert\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.497900 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.497855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f257daa1-126f-4549-a16f-4a5967506834-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.497900 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.497884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/814798a3-c8db-4727-920f-dfca65127d18-serving-cert\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.505621 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.505603 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l5hq\" (UniqueName: \"kubernetes.io/projected/f257daa1-126f-4549-a16f-4a5967506834-kube-api-access-2l5hq\") pod \"kube-storage-version-migrator-operator-7878f4b88c-zkbnp\" (UID: \"f257daa1-126f-4549-a16f-4a5967506834\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.505857 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.505839 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvq6l\" (UniqueName: \"kubernetes.io/projected/30c90e46-5199-4d39-8858-910139a1cbcf-kube-api-access-pvq6l\") pod \"service-ca-operator-784f97bcf6-ds9bz\" (UID: \"30c90e46-5199-4d39-8858-910139a1cbcf\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.506232 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.506217 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48fk\" (UniqueName: \"kubernetes.io/projected/814798a3-c8db-4727-920f-dfca65127d18-kube-api-access-p48fk\") pod \"console-operator-5ccd6b445f-9j44d\" (UID: \"814798a3-c8db-4727-920f-dfca65127d18\") " pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.507982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.507968 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" Aug 04 15:11:18.562800 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.562765 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" Aug 04 15:11:18.584520 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.584407 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:18.674014 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.673972 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b"] Aug 04 15:11:18.679449 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:18.679396 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1eb654_576c_44bd_ab10_f7a2ed3d75e3.slice/crio-bb93083069ccb37f8281bc9531749e082c4a7dec52bfa77a118c48a4218136b9 WatchSource:0}: Error finding container bb93083069ccb37f8281bc9531749e082c4a7dec52bfa77a118c48a4218136b9: Status 404 returned error can't find the container with id bb93083069ccb37f8281bc9531749e082c4a7dec52bfa77a118c48a4218136b9 Aug 04 15:11:18.700109 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.699845 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp"] Aug 04 15:11:18.704936 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:18.704905 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf257daa1_126f_4549_a16f_4a5967506834.slice/crio-0e10e33637bb874fa14db75e99d5c5ff68739250728a264e4e1d25404c7e9fc5 WatchSource:0}: Error finding container 0e10e33637bb874fa14db75e99d5c5ff68739250728a264e4e1d25404c7e9fc5: Status 404 returned error can't find the container with id 0e10e33637bb874fa14db75e99d5c5ff68739250728a264e4e1d25404c7e9fc5 Aug 04 15:11:18.744362 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.744327 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz"] Aug 04 15:11:18.747043 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:18.747011 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c90e46_5199_4d39_8858_910139a1cbcf.slice/crio-fbde8d15c91baafcd34299376d6de9b4204b2ca8e8d5a61d2cb5840728dca1f3 WatchSource:0}: Error finding container fbde8d15c91baafcd34299376d6de9b4204b2ca8e8d5a61d2cb5840728dca1f3: Status 404 returned error can't find the container with id fbde8d15c91baafcd34299376d6de9b4204b2ca8e8d5a61d2cb5840728dca1f3 Aug 04 15:11:18.775844 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.775817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-9j44d"] Aug 04 15:11:18.798956 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.798926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:18.799046 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.799025 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:18.799090 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.799080 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls podName:f2fb7ca7-9668-4869-a3df-5022e5c62cf1 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:19.799063346 +0000 UTC m=+45.153151084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-sxq7z" (UID: "f2fb7ca7-9668-4869-a3df-5022e5c62cf1") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:18.900225 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.900187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.900374 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.900239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:18.900374 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:18.900293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:18.900374 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.900337 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:11:18.900545 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.900398 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 15:11:18.900545 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.900407 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:19.900390736 +0000 UTC m=+45.254478475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : secret "router-metrics-certs-default" not found Aug 04 15:11:18.900545 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.900422 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:19.900416169 +0000 UTC m=+45.254503905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : configmap references non-existent config key: service-ca.crt Aug 04 15:11:18.900545 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:18.900432 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls podName:2af465a1-b59e-4dc0-99c2-828508ac8a4c nodeName:}" failed. No retries permitted until 2026-08-04 15:11:19.900427091 +0000 UTC m=+45.254514828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-wj5cl" (UID: "2af465a1-b59e-4dc0-99c2-828508ac8a4c") : secret "samples-operator-tls" not found Aug 04 15:11:19.482318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.482281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" event={"ID":"814798a3-c8db-4727-920f-dfca65127d18","Type":"ContainerStarted","Data":"17758aeb1b786b25b65c3b47a2c4bfa3be935976dcf6778281e30168a5c54a8f"} Aug 04 15:11:19.484091 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.484021 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" event={"ID":"f257daa1-126f-4549-a16f-4a5967506834","Type":"ContainerStarted","Data":"0e10e33637bb874fa14db75e99d5c5ff68739250728a264e4e1d25404c7e9fc5"} Aug 04 15:11:19.486182 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.486155 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" event={"ID":"dc1eb654-576c-44bd-ab10-f7a2ed3d75e3","Type":"ContainerStarted","Data":"bb93083069ccb37f8281bc9531749e082c4a7dec52bfa77a118c48a4218136b9"} Aug 04 15:11:19.488051 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.488024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" event={"ID":"30c90e46-5199-4d39-8858-910139a1cbcf","Type":"ContainerStarted","Data":"fbde8d15c91baafcd34299376d6de9b4204b2ca8e8d5a61d2cb5840728dca1f3"} Aug 04 15:11:19.809031 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.808939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:19.809192 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:19.809143 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:19.809253 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:19.809210 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls podName:f2fb7ca7-9668-4869-a3df-5022e5c62cf1 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:21.809189576 +0000 UTC m=+47.163277316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-sxq7z" (UID: "f2fb7ca7-9668-4869-a3df-5022e5c62cf1") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.910031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.910144 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:19.910236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:19.910379 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:19.910455 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:21.910435556 +0000 UTC m=+47.264523295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : secret "router-metrics-certs-default" not found Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:19.910529 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:21.910518218 +0000 UTC m=+47.264605959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : configmap references non-existent config key: service-ca.crt Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:19.910603 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 15:11:19.910970 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:19.910634 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls podName:2af465a1-b59e-4dc0-99c2-828508ac8a4c nodeName:}" failed. No retries permitted until 2026-08-04 15:11:21.910623912 +0000 UTC m=+47.264711651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-wj5cl" (UID: "2af465a1-b59e-4dc0-99c2-828508ac8a4c") : secret "samples-operator-tls" not found Aug 04 15:11:21.262110 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:21.262023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:21.360399 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:21.360365 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:21.830560 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:21.830524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:21.830762 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:21.830739 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:21.830830 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:21.830811 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls podName:f2fb7ca7-9668-4869-a3df-5022e5c62cf1 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:25.830790028 +0000 UTC m=+51.184877767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-sxq7z" (UID: "f2fb7ca7-9668-4869-a3df-5022e5c62cf1") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:21.931565 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:21.931530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:21.931739 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:21.931634 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:21.931739 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:21.931684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:21.931739 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:21.931695 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 15:11:21.931842 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:21.931763 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls podName:2af465a1-b59e-4dc0-99c2-828508ac8a4c nodeName:}" failed. No retries permitted until 2026-08-04 15:11:25.931744924 +0000 UTC m=+51.285832678 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-wj5cl" (UID: "2af465a1-b59e-4dc0-99c2-828508ac8a4c") : secret "samples-operator-tls" not found Aug 04 15:11:21.931842 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:21.931785 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:25.931775332 +0000 UTC m=+51.285863072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : configmap references non-existent config key: service-ca.crt Aug 04 15:11:21.931842 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:21.931783 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:11:21.931941 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:21.931848 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:25.931833705 +0000 UTC m=+51.285921442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : secret "router-metrics-certs-default" not found Aug 04 15:11:22.247590 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:22.247533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:22.616639 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:22.616604 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:24.388855 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:24.388818 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:24.560249 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:24.560184 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" podStartSLOduration=0.859551113 podStartE2EDuration="6.560164722s" totalImagesPullingTime="5.700613609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:24.560015999 +0000 UTC m=+49.914103760" watchObservedRunningTime="2026-08-04 15:11:24.560164722 +0000 UTC m=+49.914252481" Aug 04 15:11:24.684529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:24.684448 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:24.782589 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:24.782536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:24.883880 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:24.883849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:25.526316 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.526280 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" event={"ID":"6d83eac4-1c69-43f5-9f49-7e4b6cf1a92e","Type":"ContainerStarted","Data":"b119f6c4942614290bcd2e1bd4e3ea1b22c03ef5922d65c12071bd6acc1ec3b6"} Aug 04 15:11:25.528321 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.528286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-z5kg4" event={"ID":"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26","Type":"ContainerStarted","Data":"5c6221a82b8d18d1bf54169fd1de40bc3bd2bc8ab9468b01d71562b186c2af14"} Aug 04 15:11:25.530381 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.530310 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" event={"ID":"30c90e46-5199-4d39-8858-910139a1cbcf","Type":"ContainerStarted","Data":"d2a8392049d841c6446ccd7231dbf1dba209cfb9be4632dea522ea28667e6b9b"} Aug 04 15:11:25.532162 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.532140 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/0.log" Aug 04 15:11:25.532274 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.532181 2559 generic.go:356] "Generic (PLEG): container finished" podID="814798a3-c8db-4727-920f-dfca65127d18" containerID="d121617d6102966426b554fd3b83e686f2cd278da63a7f0356adbdaaaf15614d" exitCode=255 Aug 04 15:11:25.532346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.532267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" event={"ID":"814798a3-c8db-4727-920f-dfca65127d18","Type":"ContainerDied","Data":"d121617d6102966426b554fd3b83e686f2cd278da63a7f0356adbdaaaf15614d"} Aug 04 15:11:25.532427 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.532412 2559 scope.go:117] "RemoveContainer" containerID="d121617d6102966426b554fd3b83e686f2cd278da63a7f0356adbdaaaf15614d" Aug 04 15:11:25.536229 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.536209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" event={"ID":"f257daa1-126f-4549-a16f-4a5967506834","Type":"ContainerStarted","Data":"e42938210aec6c2b4d0fe9160d242fd6eeca10bf0cf8d905c05d1557e3af78f7"} Aug 04 15:11:25.540843 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.540819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-k789b" event={"ID":"dc1eb654-576c-44bd-ab10-f7a2ed3d75e3","Type":"ContainerStarted","Data":"d626fafd9b6835923aa98f37b57d45fc51c179e654bf5e0e62acf3fb706137e1"} Aug 04 15:11:25.560066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.560043 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:25.561347 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.561327 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mg2d6"] Aug 04 15:11:25.565164 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.565147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.570684 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.570658 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Aug 04 15:11:25.572702 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.572681 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Aug 04 15:11:25.581542 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.581525 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ssq9w\"" Aug 04 15:11:25.606109 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.605864 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-94v75" podStartSLOduration=2.679740266 podStartE2EDuration="8.605843857s" totalImagesPullingTime="5.926103591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:25.603382045 +0000 UTC m=+50.957469803" watchObservedRunningTime="2026-08-04 15:11:25.605843857 +0000 UTC m=+50.959931617" Aug 04 15:11:25.641928 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.641898 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mg2d6"] Aug 04 15:11:25.667808 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.667780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.667895 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.667809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7z9t\" (UniqueName: \"kubernetes.io/projected/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-api-access-s7z9t\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.667895 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.667828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-data-volume\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.667895 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.667847 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.668056 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.667914 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-crio-socket\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.712394 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.712332 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-z5kg4" podStartSLOduration=2.738828127 podStartE2EDuration="8.71231304s" totalImagesPullingTime="5.973484913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:25.711398084 +0000 UTC m=+51.065485843" watchObservedRunningTime="2026-08-04 15:11:25.71231304 +0000 UTC m=+51.066400801" Aug 04 15:11:25.726456 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.726417 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:25.732346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.732300 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" podStartSLOduration=2.041148964 podStartE2EDuration="7.732283681s" totalImagesPullingTime="5.691134717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:25.732034642 +0000 UTC m=+51.086122405" watchObservedRunningTime="2026-08-04 15:11:25.732283681 +0000 UTC m=+51.086371443" Aug 04 15:11:25.757810 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.757757 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" podStartSLOduration=2.11741612 podStartE2EDuration="7.757742231s" totalImagesPullingTime="5.640326111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:25.756854715 +0000 UTC m=+51.110942471" watchObservedRunningTime="2026-08-04 15:11:25.757742231 +0000 UTC m=+51.111829989" Aug 04 15:11:25.769259 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.769226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7z9t\" (UniqueName: \"kubernetes.io/projected/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-api-access-s7z9t\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.769395 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.769278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-data-volume\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.769395 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.769305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.769395 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.769380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-crio-socket\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.769558 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.769541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.769681 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.769666 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 15:11:25.769739 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.769729 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:26.269710823 +0000 UTC m=+51.623798578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : configmap "kube-rbac-proxy" not found Aug 04 15:11:25.770456 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.770430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-data-volume\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.770619 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.770537 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 15:11:25.770619 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.770569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-crio-socket\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.770619 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.770617 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:26.270600718 +0000 UTC m=+51.624688467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : secret "insights-runtime-extractor-tls" not found Aug 04 15:11:25.783226 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.783195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7z9t\" (UniqueName: \"kubernetes.io/projected/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-api-access-s7z9t\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:25.842479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.842440 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:25.870727 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.870695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:25.870885 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.870838 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Aug 04 15:11:25.870885 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.870855 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bb959c8b7-dv5dw: secret "image-registry-tls" not found Aug 04 15:11:25.870983 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.870888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:25.870983 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.870910 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls podName:3c793f63-8a0d-40e5-8f6a-61b6ce8a7931 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:41.870893511 +0000 UTC m=+67.224981251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls") pod "image-registry-7bb959c8b7-dv5dw" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931") : secret "image-registry-tls" not found Aug 04 15:11:25.870983 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.870942 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:25.870983 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.870960 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Aug 04 15:11:25.871181 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.871037 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls podName:edc4b4c4-6033-4d2d-87ff-c1be81027a01 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:41.871004398 +0000 UTC m=+67.225092142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls") pod "dns-default-2bng9" (UID: "edc4b4c4-6033-4d2d-87ff-c1be81027a01") : secret "dns-default-metrics-tls" not found Aug 04 15:11:25.871181 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.871094 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:25.871181 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.871161 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls podName:f2fb7ca7-9668-4869-a3df-5022e5c62cf1 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:33.871145861 +0000 UTC m=+59.225233606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-sxq7z" (UID: "f2fb7ca7-9668-4869-a3df-5022e5c62cf1") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:25.942511 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.942470 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:25.971607 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.971563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:25.971778 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.971648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:25.971778 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:25.971681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:25.971778 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.971709 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Aug 04 15:11:25.971778 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.971762 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls podName:2af465a1-b59e-4dc0-99c2-828508ac8a4c nodeName:}" failed. No retries permitted until 2026-08-04 15:11:33.971748246 +0000 UTC m=+59.325835983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-wj5cl" (UID: "2af465a1-b59e-4dc0-99c2-828508ac8a4c") : secret "samples-operator-tls" not found Aug 04 15:11:25.971974 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.971786 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:11:25.971974 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.971811 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:33.971800518 +0000 UTC m=+59.325888256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : configmap references non-existent config key: service-ca.crt Aug 04 15:11:25.971974 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:25.971831 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:33.971820292 +0000 UTC m=+59.325908032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : secret "router-metrics-certs-default" not found Aug 04 15:11:26.080384 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.080350 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:26.175757 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.175721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:26.275418 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.275345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:26.275418 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.275385 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:26.275654 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:26.275467 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 15:11:26.275654 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:26.275533 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:27.27551571 +0000 UTC m=+52.629603451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : configmap "kube-rbac-proxy" not found Aug 04 15:11:26.275654 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:26.275583 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 15:11:26.275654 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:26.275643 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:27.275627006 +0000 UTC m=+52.629714762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : secret "insights-runtime-extractor-tls" not found Aug 04 15:11:26.306279 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.306246 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:26.328675 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.328636 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp"] Aug 04 15:11:26.330867 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.330847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" Aug 04 15:11:26.333754 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.333731 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Aug 04 15:11:26.333888 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.333848 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Aug 04 15:11:26.333931 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.333854 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4z8b9\"" Aug 04 15:11:26.346463 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.346439 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp"] Aug 04 15:11:26.376544 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.376516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg8c6\" (UniqueName: \"kubernetes.io/projected/cc582a60-5a1e-40a4-bcda-8c02ed5629cd-kube-api-access-dg8c6\") pod \"migrator-68bc745d87-6lxxp\" (UID: \"cc582a60-5a1e-40a4-bcda-8c02ed5629cd\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" Aug 04 15:11:26.407986 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.407956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:26.477434 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.477396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg8c6\" (UniqueName: \"kubernetes.io/projected/cc582a60-5a1e-40a4-bcda-8c02ed5629cd-kube-api-access-dg8c6\") pod \"migrator-68bc745d87-6lxxp\" (UID: \"cc582a60-5a1e-40a4-bcda-8c02ed5629cd\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" Aug 04 15:11:26.487096 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.487069 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg8c6\" (UniqueName: \"kubernetes.io/projected/cc582a60-5a1e-40a4-bcda-8c02ed5629cd-kube-api-access-dg8c6\") pod \"migrator-68bc745d87-6lxxp\" (UID: \"cc582a60-5a1e-40a4-bcda-8c02ed5629cd\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" Aug 04 15:11:26.508050 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.508007 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:26.546430 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.546361 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/1.log" Aug 04 15:11:26.546808 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.546782 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/0.log" Aug 04 15:11:26.546845 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.546812 2559 generic.go:356] "Generic (PLEG): container finished" podID="814798a3-c8db-4727-920f-dfca65127d18" containerID="ccb21bbc2ff08213837de93096c6c5258c0f945c3dad39b3bd54f41428855d6a" exitCode=255 Aug 04 15:11:26.546942 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.546910 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" event={"ID":"814798a3-c8db-4727-920f-dfca65127d18","Type":"ContainerDied","Data":"ccb21bbc2ff08213837de93096c6c5258c0f945c3dad39b3bd54f41428855d6a"} Aug 04 15:11:26.547049 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.546967 2559 scope.go:117] "RemoveContainer" containerID="d121617d6102966426b554fd3b83e686f2cd278da63a7f0356adbdaaaf15614d" Aug 04 15:11:26.547111 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.547093 2559 scope.go:117] "RemoveContainer" containerID="ccb21bbc2ff08213837de93096c6c5258c0f945c3dad39b3bd54f41428855d6a" Aug 04 15:11:26.547328 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:26.547301 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-9j44d_openshift-console-operator(814798a3-c8db-4727-920f-dfca65127d18)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" podUID="814798a3-c8db-4727-920f-dfca65127d18" Aug 04 15:11:26.624945 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.624912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:26.640088 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.640068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" Aug 04 15:11:26.761565 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.761537 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp"] Aug 04 15:11:26.763619 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:26.763594 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc582a60_5a1e_40a4_bcda_8c02ed5629cd.slice/crio-e4016da2d7ac2dbf562685620e0c66ea5b435a75d87ff8a39fa530fe834ca7ce WatchSource:0}: Error finding container e4016da2d7ac2dbf562685620e0c66ea5b435a75d87ff8a39fa530fe834ca7ce: Status 404 returned error can't find the container with id e4016da2d7ac2dbf562685620e0c66ea5b435a75d87ff8a39fa530fe834ca7ce Aug 04 15:11:26.866884 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:26.866818 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j8h9m_7632fc34-ddbf-4d83-8048-0a9d86cdf1a0/dns-node-resolver/0.log" Aug 04 15:11:27.286438 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:27.286404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:27.286643 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:27.286456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:27.286643 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:27.286509 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 15:11:27.286643 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:27.286564 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:29.286548393 +0000 UTC m=+54.640636134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : configmap "kube-rbac-proxy" not found Aug 04 15:11:27.286643 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:27.286635 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 15:11:27.286838 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:27.286696 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:29.286677727 +0000 UTC m=+54.640765467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : secret "insights-runtime-extractor-tls" not found Aug 04 15:11:27.551546 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:27.551467 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/1.log" Aug 04 15:11:27.551994 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:27.551896 2559 scope.go:117] "RemoveContainer" containerID="ccb21bbc2ff08213837de93096c6c5258c0f945c3dad39b3bd54f41428855d6a" Aug 04 15:11:27.552115 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:27.552090 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-9j44d_openshift-console-operator(814798a3-c8db-4727-920f-dfca65127d18)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" podUID="814798a3-c8db-4727-920f-dfca65127d18" Aug 04 15:11:27.552820 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:27.552791 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" event={"ID":"cc582a60-5a1e-40a4-bcda-8c02ed5629cd","Type":"ContainerStarted","Data":"e4016da2d7ac2dbf562685620e0c66ea5b435a75d87ff8a39fa530fe834ca7ce"} Aug 04 15:11:27.666571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:27.666539 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lk76r_fa1d177c-d3ac-425d-a836-58dc1dc7b425/node-ca/0.log" Aug 04 15:11:28.160919 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.160891 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-86crf"] Aug 04 15:11:28.163139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.163124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.166635 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.166603 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Aug 04 15:11:28.166790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.166773 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-sdgg5\"" Aug 04 15:11:28.167180 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.167165 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Aug 04 15:11:28.167235 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.167222 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Aug 04 15:11:28.167784 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.167771 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Aug 04 15:11:28.173331 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.173309 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-86crf"] Aug 04 15:11:28.194909 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.194882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-signing-cabundle\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.195038 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.195007 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-signing-key\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.195103 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.195067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x278l\" (UniqueName: \"kubernetes.io/projected/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-kube-api-access-x278l\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.207864 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.207826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:28.271274 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.271242 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mlgv6_c389923e-0891-45d8-b662-1700d98d597b/serve-healthcheck-canary/0.log" Aug 04 15:11:28.295791 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.295759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x278l\" (UniqueName: \"kubernetes.io/projected/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-kube-api-access-x278l\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.295906 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.295859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-signing-cabundle\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.295997 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.295982 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-signing-key\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.296640 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.296621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-signing-cabundle\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.296932 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.296914 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:28.298548 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.298525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-signing-key\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.305234 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.305213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x278l\" (UniqueName: \"kubernetes.io/projected/9b0e3376-2572-4d2b-8aa6-d84da324fcc7-kube-api-access-x278l\") pod \"service-ca-98c88fbb5-86crf\" (UID: \"9b0e3376-2572-4d2b-8aa6-d84da324fcc7\") " pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.472211 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.472176 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-86crf" Aug 04 15:11:28.484263 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.484226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:28.559531 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.559489 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" event={"ID":"cc582a60-5a1e-40a4-bcda-8c02ed5629cd","Type":"ContainerStarted","Data":"1f004b8bb6473700afd42fa7e7da823b6a1fd72bb302311cc996ea8ebd69410d"} Aug 04 15:11:28.559929 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.559553 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" event={"ID":"cc582a60-5a1e-40a4-bcda-8c02ed5629cd","Type":"ContainerStarted","Data":"37bd66d859dc6b352f12ddc22f0f9445d6b64d18dd512cf9091977aa7d823cb0"} Aug 04 15:11:28.581539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.581490 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-6lxxp" podStartSLOduration=1.3416337870000001 podStartE2EDuration="2.581468923s" totalImagesPullingTime="1.239835136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:28.581416809 +0000 UTC m=+53.935504567" watchObservedRunningTime="2026-08-04 15:11:28.581468923 +0000 UTC m=+53.935556681" Aug 04 15:11:28.585508 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.585483 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:28.585653 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.585528 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:28.585848 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.585837 2559 scope.go:117] "RemoveContainer" containerID="ccb21bbc2ff08213837de93096c6c5258c0f945c3dad39b3bd54f41428855d6a" Aug 04 15:11:28.585995 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:28.585980 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-9j44d_openshift-console-operator(814798a3-c8db-4727-920f-dfca65127d18)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" podUID="814798a3-c8db-4727-920f-dfca65127d18" Aug 04 15:11:28.596483 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.596363 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-86crf"] Aug 04 15:11:28.599052 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:28.599032 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0e3376_2572_4d2b_8aa6_d84da324fcc7.slice/crio-27ea921f5e6458efbcad07abea3394fb81d105424029f734f076103d8f7002c0 WatchSource:0}: Error finding container 27ea921f5e6458efbcad07abea3394fb81d105424029f734f076103d8f7002c0: Status 404 returned error can't find the container with id 27ea921f5e6458efbcad07abea3394fb81d105424029f734f076103d8f7002c0 Aug 04 15:11:28.711812 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.711782 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:28.825111 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:28.825010 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:29.304779 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:29.304745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:29.305048 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:29.304789 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:29.305048 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:29.304872 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 15:11:29.305048 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:29.304898 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 15:11:29.305048 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:29.304939 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:33.304921777 +0000 UTC m=+58.659009518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : configmap "kube-rbac-proxy" not found Aug 04 15:11:29.305201 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:29.305073 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:33.305058569 +0000 UTC m=+58.659146306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : secret "insights-runtime-extractor-tls" not found Aug 04 15:11:29.513055 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:29.513023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:29.564999 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:29.564902 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-86crf" event={"ID":"9b0e3376-2572-4d2b-8aa6-d84da324fcc7","Type":"ContainerStarted","Data":"5d0be6798d4adb7b998581b4272548cf1473bbb12957ef46cf3e70901bac4d8e"} Aug 04 15:11:29.564999 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:29.564959 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-86crf" event={"ID":"9b0e3376-2572-4d2b-8aa6-d84da324fcc7","Type":"ContainerStarted","Data":"27ea921f5e6458efbcad07abea3394fb81d105424029f734f076103d8f7002c0"} Aug 04 15:11:29.588406 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:29.588359 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-86crf" podStartSLOduration=1.588343591 podStartE2EDuration="1.588343591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:29.588125743 +0000 UTC m=+54.942213515" watchObservedRunningTime="2026-08-04 15:11:29.588343591 +0000 UTC m=+54.942431346" Aug 04 15:11:30.424309 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:30.424272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:30.529928 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:30.529893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:30.635740 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:30.635697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:30.713357 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:30.713324 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:30.822547 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:30.822513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:31.035060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:31.034967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:32.030691 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:32.030647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:32.465790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:32.465754 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:32.627703 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:32.627648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:32.822564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:32.822475 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:33.018749 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:33.018713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:33.344007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:33.343954 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:33.344007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:33.344009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:33.344451 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:33.344090 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Aug 04 15:11:33.344451 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:33.344094 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Aug 04 15:11:33.344451 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:33.344155 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:41.344140013 +0000 UTC m=+66.698227750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : configmap "kube-rbac-proxy" not found Aug 04 15:11:33.344451 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:33.344195 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls podName:b3f2ae45-d92a-4422-9ecb-42a0c822daf6 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:41.344181603 +0000 UTC m=+66.698269340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mg2d6" (UID: "b3f2ae45-d92a-4422-9ecb-42a0c822daf6") : secret "insights-runtime-extractor-tls" not found Aug 04 15:11:33.948299 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:33.948260 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:33.948477 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:33.948378 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:33.948477 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:33.948439 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls podName:f2fb7ca7-9668-4869-a3df-5022e5c62cf1 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:49.948421979 +0000 UTC m=+75.302509716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-sxq7z" (UID: "f2fb7ca7-9668-4869-a3df-5022e5c62cf1") : secret "cluster-monitoring-operator-tls" not found Aug 04 15:11:34.049296 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.049248 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:34.049471 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:34.049413 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Aug 04 15:11:34.049524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.049420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:34.049524 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:34.049486 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:50.049466896 +0000 UTC m=+75.403554646 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : secret "router-metrics-certs-default" not found Aug 04 15:11:34.049639 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:34.049523 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle podName:c330d986-f440-4bb4-b152-bdb21af2925e nodeName:}" failed. No retries permitted until 2026-08-04 15:11:50.049505593 +0000 UTC m=+75.403593333 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle") pod "router-default-59fcd9fd6d-l6gbg" (UID: "c330d986-f440-4bb4-b152-bdb21af2925e") : configmap references non-existent config key: service-ca.crt Aug 04 15:11:34.049681 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.049653 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:34.052027 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.052005 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2af465a1-b59e-4dc0-99c2-828508ac8a4c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-wj5cl\" (UID: \"2af465a1-b59e-4dc0-99c2-828508ac8a4c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:34.336972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.336889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" Aug 04 15:11:34.493753 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.493635 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl"] Aug 04 15:11:34.594635 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.594548 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" event={"ID":"2af465a1-b59e-4dc0-99c2-828508ac8a4c","Type":"ContainerStarted","Data":"20878c1abac0a38c08ac9ce08755c276bdaae54be499c6a11839dfe2bd2201fe"} Aug 04 15:11:34.922150 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:34.922068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:35.010420 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:35.010378 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:35.089507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:35.087028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:35.439511 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:35.439481 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-646lg" Aug 04 15:11:36.244671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:36.244631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:36.345051 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:36.345019 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:36.445562 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:36.445491 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:36.637478 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:36.637439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:36.895083 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:36.895008 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:37.039851 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:37.039814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:37.606767 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:37.606734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" event={"ID":"2af465a1-b59e-4dc0-99c2-828508ac8a4c","Type":"ContainerStarted","Data":"147971d704f4e00ab19ca3e9c2f8286df4dd4690f223b238d6126d18e21c87c0"} Aug 04 15:11:37.607149 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:37.606793 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" event={"ID":"2af465a1-b59e-4dc0-99c2-828508ac8a4c","Type":"ContainerStarted","Data":"5430f988563e2b50e8fd731f990778e3c3b18dd417c27bdaa2e2bea514ddb9b0"} Aug 04 15:11:37.628642 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:37.628592 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-wj5cl" podStartSLOduration=17.621843825 podStartE2EDuration="19.628556944s" totalImagesPullingTime="2.006713119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:37.627967055 +0000 UTC m=+62.982054815" watchObservedRunningTime="2026-08-04 15:11:37.628556944 +0000 UTC m=+62.982644702" Aug 04 15:11:38.222479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:38.222441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:38.401465 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:38.401429 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:38.500765 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:38.500685 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:38.625877 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:38.625842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:38.847976 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:38.847891 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:39.022887 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:39.022859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:39.410944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:39.410912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:39.570671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:39.570629 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:39.839252 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:39.839219 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:40.234630 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.234599 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:40.282198 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.282170 2559 scope.go:117] "RemoveContainer" containerID="ccb21bbc2ff08213837de93096c6c5258c0f945c3dad39b3bd54f41428855d6a" Aug 04 15:11:40.337067 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.337033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:40.434871 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.434835 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:40.504119 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.504021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:40.618731 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.618702 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:11:40.619122 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.619103 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/1.log" Aug 04 15:11:40.619230 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.619140 2559 generic.go:356] "Generic (PLEG): container finished" podID="814798a3-c8db-4727-920f-dfca65127d18" containerID="62f8075d0e02d4c1b4877efa25868ee54d8edf57b2dd805bc6f7af03b95738aa" exitCode=255 Aug 04 15:11:40.619230 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.619211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" event={"ID":"814798a3-c8db-4727-920f-dfca65127d18","Type":"ContainerDied","Data":"62f8075d0e02d4c1b4877efa25868ee54d8edf57b2dd805bc6f7af03b95738aa"} Aug 04 15:11:40.619332 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.619263 2559 scope.go:117] "RemoveContainer" containerID="ccb21bbc2ff08213837de93096c6c5258c0f945c3dad39b3bd54f41428855d6a" Aug 04 15:11:40.619590 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.619565 2559 scope.go:117] "RemoveContainer" containerID="62f8075d0e02d4c1b4877efa25868ee54d8edf57b2dd805bc6f7af03b95738aa" Aug 04 15:11:40.619820 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:40.619794 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-9j44d_openshift-console-operator(814798a3-c8db-4727-920f-dfca65127d18)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" podUID="814798a3-c8db-4727-920f-dfca65127d18" Aug 04 15:11:40.637382 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.637362 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:40.908644 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.908603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:40.911810 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.911788 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Aug 04 15:11:40.921197 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.921179 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d26e3571-af26-4a3d-ab3f-0800c3b1f66b-metrics-certs\") pod \"network-metrics-daemon-w2npp\" (UID: \"d26e3571-af26-4a3d-ab3f-0800c3b1f66b\") " pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:40.995344 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:40.995312 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8gfq\"" Aug 04 15:11:41.003530 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.003505 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w2npp" Aug 04 15:11:41.010073 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.010057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:41.013937 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.013908 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Aug 04 15:11:41.022792 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.022772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e629fc59-eefc-49cd-9bb5-9a329b04633b-original-pull-secret\") pod \"global-pull-secret-syncer-6nqlc\" (UID: \"e629fc59-eefc-49cd-9bb5-9a329b04633b\") " pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:41.111152 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.111084 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:41.113364 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.113341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcld\" (UniqueName: \"kubernetes.io/projected/86dfd1b4-818f-4c5f-b7da-63998cbf15d8-kube-api-access-bbcld\") pod \"network-check-target-kl72z\" (UID: \"86dfd1b4-818f-4c5f-b7da-63998cbf15d8\") " pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:41.128091 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.128055 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w2npp"] Aug 04 15:11:41.130737 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:41.130713 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26e3571_af26_4a3d_ab3f_0800c3b1f66b.slice/crio-b4146b453aff41755dca8117b60ce0e69d59e1489593a03e3651cafa22681c1a WatchSource:0}: Error finding container b4146b453aff41755dca8117b60ce0e69d59e1489593a03e3651cafa22681c1a: Status 404 returned error can't find the container with id b4146b453aff41755dca8117b60ce0e69d59e1489593a03e3651cafa22681c1a Aug 04 15:11:41.150681 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.150659 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:41.195778 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.195755 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-l7r4m\"" Aug 04 15:11:41.203108 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.203091 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:41.301589 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.301548 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6nqlc" Aug 04 15:11:41.335284 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.335242 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kl72z"] Aug 04 15:11:41.337601 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:41.337545 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dfd1b4_818f_4c5f_b7da_63998cbf15d8.slice/crio-81dfe90b94715baf0f47c966ee6456b77a4558b0571aa5d38e7cb170c71fc757 WatchSource:0}: Error finding container 81dfe90b94715baf0f47c966ee6456b77a4558b0571aa5d38e7cb170c71fc757: Status 404 returned error can't find the container with id 81dfe90b94715baf0f47c966ee6456b77a4558b0571aa5d38e7cb170c71fc757 Aug 04 15:11:41.413477 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.413451 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:41.413652 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.413491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:41.413946 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.413928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:41.415631 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.415554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b3f2ae45-d92a-4422-9ecb-42a0c822daf6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mg2d6\" (UID: \"b3f2ae45-d92a-4422-9ecb-42a0c822daf6\") " pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:41.432786 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.432657 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6nqlc"] Aug 04 15:11:41.435105 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:41.435082 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode629fc59_eefc_49cd_9bb5_9a329b04633b.slice/crio-6b03288171b97ad9117e0c5ce9700c5dc295248cc381ba8ebda719ad7f6b04f4 WatchSource:0}: Error finding container 6b03288171b97ad9117e0c5ce9700c5dc295248cc381ba8ebda719ad7f6b04f4: Status 404 returned error can't find the container with id 6b03288171b97ad9117e0c5ce9700c5dc295248cc381ba8ebda719ad7f6b04f4 Aug 04 15:11:41.498072 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.498046 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ssq9w\"" Aug 04 15:11:41.505241 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.505222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mg2d6" Aug 04 15:11:41.624017 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.623955 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:11:41.625125 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.625088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6nqlc" event={"ID":"e629fc59-eefc-49cd-9bb5-9a329b04633b","Type":"ContainerStarted","Data":"6b03288171b97ad9117e0c5ce9700c5dc295248cc381ba8ebda719ad7f6b04f4"} Aug 04 15:11:41.626468 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.626446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kl72z" event={"ID":"86dfd1b4-818f-4c5f-b7da-63998cbf15d8","Type":"ContainerStarted","Data":"08a90e6683a34725fb6c1cdd7c1bcfa18dc8412aab8e106e054f120481fd3446"} Aug 04 15:11:41.626609 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.626487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kl72z" event={"ID":"86dfd1b4-818f-4c5f-b7da-63998cbf15d8","Type":"ContainerStarted","Data":"81dfe90b94715baf0f47c966ee6456b77a4558b0571aa5d38e7cb170c71fc757"} Aug 04 15:11:41.626609 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.626512 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:11:41.627607 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.627565 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w2npp" event={"ID":"d26e3571-af26-4a3d-ab3f-0800c3b1f66b","Type":"ContainerStarted","Data":"b4146b453aff41755dca8117b60ce0e69d59e1489593a03e3651cafa22681c1a"} Aug 04 15:11:41.630973 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.630952 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mg2d6"] Aug 04 15:11:41.632428 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:41.632406 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f2ae45_d92a_4422_9ecb_42a0c822daf6.slice/crio-a0fb5228242cbcb9d6dc0ca74263321dbd704b7ff25469bbac98e1f358423ba0 WatchSource:0}: Error finding container a0fb5228242cbcb9d6dc0ca74263321dbd704b7ff25469bbac98e1f358423ba0: Status 404 returned error can't find the container with id a0fb5228242cbcb9d6dc0ca74263321dbd704b7ff25469bbac98e1f358423ba0 Aug 04 15:11:41.648607 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.648537 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kl72z" podStartSLOduration=66.648518825 podStartE2EDuration="1m6.648518825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:41.647456174 +0000 UTC m=+67.001543944" watchObservedRunningTime="2026-08-04 15:11:41.648518825 +0000 UTC m=+67.002606585" Aug 04 15:11:41.772482 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.772444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:41.879784 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.879658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:41.918345 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.918306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:41.918772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.918420 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:41.920914 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.920891 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edc4b4c4-6033-4d2d-87ff-c1be81027a01-metrics-tls\") pod \"dns-default-2bng9\" (UID: \"edc4b4c4-6033-4d2d-87ff-c1be81027a01\") " pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:41.921018 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:41.920974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"image-registry-7bb959c8b7-dv5dw\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:42.107184 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.107155 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cqks7\"" Aug 04 15:11:42.114232 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.114205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:42.144384 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.144310 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xckv6\"" Aug 04 15:11:42.151652 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.151626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:42.471048 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.470947 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2bng9"] Aug 04 15:11:42.476790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.476703 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bb959c8b7-dv5dw"] Aug 04 15:11:42.477370 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:42.477333 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc4b4c4_6033_4d2d_87ff_c1be81027a01.slice/crio-dd766a5f37c3c34bef2390cb42d8d1354bdbe1e3916f4b9b2d4066c1643c404d WatchSource:0}: Error finding container dd766a5f37c3c34bef2390cb42d8d1354bdbe1e3916f4b9b2d4066c1643c404d: Status 404 returned error can't find the container with id dd766a5f37c3c34bef2390cb42d8d1354bdbe1e3916f4b9b2d4066c1643c404d Aug 04 15:11:42.478114 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:42.478092 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c793f63_8a0d_40e5_8f6a_61b6ce8a7931.slice/crio-12f5d6b388d9db9574778427f66773c2c77c37650d8eb437de922abe805878db WatchSource:0}: Error finding container 12f5d6b388d9db9574778427f66773c2c77c37650d8eb437de922abe805878db: Status 404 returned error can't find the container with id 12f5d6b388d9db9574778427f66773c2c77c37650d8eb437de922abe805878db Aug 04 15:11:42.635434 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.635394 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w2npp" event={"ID":"d26e3571-af26-4a3d-ab3f-0800c3b1f66b","Type":"ContainerStarted","Data":"76a57bd2e2692ebba0ca5fa57800ba9bc29e1672edbaa9d60b4d4c2a78c6789a"} Aug 04 15:11:42.635631 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.635466 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w2npp" event={"ID":"d26e3571-af26-4a3d-ab3f-0800c3b1f66b","Type":"ContainerStarted","Data":"73fa655da427f591e1efd227e90a5a0a8979ddd99dccffc69e5cf95ff2f3f528"} Aug 04 15:11:42.637068 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.637038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" event={"ID":"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931","Type":"ContainerStarted","Data":"392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782"} Aug 04 15:11:42.637190 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.637095 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:11:42.637190 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.637115 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" event={"ID":"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931","Type":"ContainerStarted","Data":"12f5d6b388d9db9574778427f66773c2c77c37650d8eb437de922abe805878db"} Aug 04 15:11:42.638775 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.638751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg2d6" event={"ID":"b3f2ae45-d92a-4422-9ecb-42a0c822daf6","Type":"ContainerStarted","Data":"460825052cf37f1f4e68e84bfd91f68d352fb389b1eeae6f912ab3174dd71e82"} Aug 04 15:11:42.638894 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.638801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg2d6" event={"ID":"b3f2ae45-d92a-4422-9ecb-42a0c822daf6","Type":"ContainerStarted","Data":"7fc5747e3a27c59d13fa3486897ed74fd5e0a0b3b1a3950dc528718b4df8c2c3"} Aug 04 15:11:42.638894 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.638830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg2d6" event={"ID":"b3f2ae45-d92a-4422-9ecb-42a0c822daf6","Type":"ContainerStarted","Data":"a0fb5228242cbcb9d6dc0ca74263321dbd704b7ff25469bbac98e1f358423ba0"} Aug 04 15:11:42.639893 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.639873 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bng9" event={"ID":"edc4b4c4-6033-4d2d-87ff-c1be81027a01","Type":"ContainerStarted","Data":"dd766a5f37c3c34bef2390cb42d8d1354bdbe1e3916f4b9b2d4066c1643c404d"} Aug 04 15:11:42.684141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.684058 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w2npp" podStartSLOduration=66.538052552 podStartE2EDuration="1m7.684042964s" totalImagesPullingTime="1.145990412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:42.656683688 +0000 UTC m=+68.010771499" watchObservedRunningTime="2026-08-04 15:11:42.684042964 +0000 UTC m=+68.038130722" Aug 04 15:11:42.684441 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:42.684421 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" podStartSLOduration=66.684414873 podStartE2EDuration="1m6.684414873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:42.682019196 +0000 UTC m=+68.036106955" watchObservedRunningTime="2026-08-04 15:11:42.684414873 +0000 UTC m=+68.038502633" Aug 04 15:11:44.654607 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:44.654500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mg2d6" event={"ID":"b3f2ae45-d92a-4422-9ecb-42a0c822daf6","Type":"ContainerStarted","Data":"40538c61ace951fe1f8dcd9297604dc47d3816c2e6e39f738c5f0bfcbba08341"} Aug 04 15:11:44.656433 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:44.656406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bng9" event={"ID":"edc4b4c4-6033-4d2d-87ff-c1be81027a01","Type":"ContainerStarted","Data":"bf1b3008228a3f58ea8b8606ec7f0f28679ea6ee4f2104b7b51f3d24d43e0484"} Aug 04 15:11:44.656556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:44.656451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2bng9" event={"ID":"edc4b4c4-6033-4d2d-87ff-c1be81027a01","Type":"ContainerStarted","Data":"8ad9932d04f9f280ff5a60812e7d196307d709cb67c1f8bc135fdcb672e1f3b7"} Aug 04 15:11:44.656556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:44.656472 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:44.680653 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:44.680611 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mg2d6" podStartSLOduration=17.786759869 podStartE2EDuration="19.680597826s" totalImagesPullingTime="1.893837957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:11:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:44.678992837 +0000 UTC m=+70.033080606" watchObservedRunningTime="2026-08-04 15:11:44.680597826 +0000 UTC m=+70.034685585" Aug 04 15:11:44.701668 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:44.701630 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2bng9" podStartSLOduration=33.816754304 podStartE2EDuration="35.701616946s" totalImagesPullingTime="1.884862642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:44.700037575 +0000 UTC m=+70.054125335" watchObservedRunningTime="2026-08-04 15:11:44.701616946 +0000 UTC m=+70.055704707" Aug 04 15:11:47.668956 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:47.668919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6nqlc" event={"ID":"e629fc59-eefc-49cd-9bb5-9a329b04633b","Type":"ContainerStarted","Data":"d9ab471e081ede417e1e1dc1eda6f4265dfc9c0fd51702bee407b764fb880b34"} Aug 04 15:11:47.693643 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:47.693559 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6nqlc" podStartSLOduration=66.39886144 podStartE2EDuration="1m11.693544858s" totalImagesPullingTime="5.294683418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 15:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:47.692964731 +0000 UTC m=+73.047052489" watchObservedRunningTime="2026-08-04 15:11:47.693544858 +0000 UTC m=+73.047632616" Aug 04 15:11:48.584725 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:48.584684 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:48.584874 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:48.584772 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:11:48.585090 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:48.585079 2559 scope.go:117] "RemoveContainer" containerID="62f8075d0e02d4c1b4877efa25868ee54d8edf57b2dd805bc6f7af03b95738aa" Aug 04 15:11:48.585246 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:48.585227 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-5ccd6b445f-9j44d_openshift-console-operator(814798a3-c8db-4727-920f-dfca65127d18)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" podUID="814798a3-c8db-4727-920f-dfca65127d18" Aug 04 15:11:49.988308 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:49.988267 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:49.990634 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:49.990614 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2fb7ca7-9668-4869-a3df-5022e5c62cf1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-sxq7z\" (UID: \"f2fb7ca7-9668-4869-a3df-5022e5c62cf1\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:50.089339 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.089308 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:50.089339 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.089347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:50.090101 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.090079 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c330d986-f440-4bb4-b152-bdb21af2925e-service-ca-bundle\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:50.091831 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.091808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c330d986-f440-4bb4-b152-bdb21af2925e-metrics-certs\") pod \"router-default-59fcd9fd6d-l6gbg\" (UID: \"c330d986-f440-4bb4-b152-bdb21af2925e\") " pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:50.124074 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.124044 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-dblpb\"" Aug 04 15:11:50.131869 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.131849 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" Aug 04 15:11:50.205933 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.205898 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk"] Aug 04 15:11:50.208947 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.208923 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj"] Aug 04 15:11:50.209090 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.209040 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.211298 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.211254 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.211446 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.211432 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd"] Aug 04 15:11:50.212392 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.212369 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-c2j8d\"" Aug 04 15:11:50.212517 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.212496 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Aug 04 15:11:50.212923 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.212901 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Aug 04 15:11:50.214238 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.214221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.215029 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.214657 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Aug 04 15:11:50.217344 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.217325 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Aug 04 15:11:50.217439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.217382 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Aug 04 15:11:50.217496 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.217468 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Aug 04 15:11:50.217595 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.217567 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Aug 04 15:11:50.217701 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.217688 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Aug 04 15:11:50.220307 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.220291 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Aug 04 15:11:50.220387 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.220290 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Aug 04 15:11:50.237186 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.236924 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj"] Aug 04 15:11:50.238527 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.238468 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx"] Aug 04 15:11:50.241039 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.241000 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:50.241691 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.241617 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk"] Aug 04 15:11:50.244304 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:50.244251 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-dockercfg-cwhcp\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cwhcp\"" type="*v1.Secret" Aug 04 15:11:50.244483 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:50.244453 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"managed-serviceaccount-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" Aug 04 15:11:50.244607 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.244511 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd"] Aug 04 15:11:50.264401 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.264377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx"] Aug 04 15:11:50.276728 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.276709 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dndng\"" Aug 04 15:11:50.277242 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.277214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:50.283627 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.283598 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:50.290881 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.290860 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/07f59c73-3208-4518-98d8-2794c26dca31-klusterlet-config\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.290982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.290896 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmlk4\" (UniqueName: \"kubernetes.io/projected/07f59c73-3208-4518-98d8-2794c26dca31-kube-api-access-kmlk4\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.290982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.290935 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-ca\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.291066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.290980 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.291066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.291011 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f59c73-3208-4518-98d8-2794c26dca31-tmp\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.291066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.291031 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb6g7\" (UniqueName: \"kubernetes.io/projected/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-kube-api-access-qb6g7\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.291160 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.291089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.291160 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.291107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-hub\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.291160 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.291135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/55a38cc2-ffaa-46be-a929-a3d7fb71250d-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-4tkkk\" (UID: \"55a38cc2-ffaa-46be-a929-a3d7fb71250d\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.291260 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.291163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.291260 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.291209 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/55a38cc2-ffaa-46be-a929-a3d7fb71250d-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-4tkkk\" (UID: \"55a38cc2-ffaa-46be-a929-a3d7fb71250d\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.298042 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.298020 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z"] Aug 04 15:11:50.300451 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:50.300427 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fb7ca7_9668_4869_a3df_5022e5c62cf1.slice/crio-e0a80ae975f51acce19f14ea79308d176b9fd3360c5e63c5c840fb86b035e2d2 WatchSource:0}: Error finding container e0a80ae975f51acce19f14ea79308d176b9fd3360c5e63c5c840fb86b035e2d2: Status 404 returned error can't find the container with id e0a80ae975f51acce19f14ea79308d176b9fd3360c5e63c5c840fb86b035e2d2 Aug 04 15:11:50.392482 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/55a38cc2-ffaa-46be-a929-a3d7fb71250d-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-4tkkk\" (UID: \"55a38cc2-ffaa-46be-a929-a3d7fb71250d\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.392663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/07f59c73-3208-4518-98d8-2794c26dca31-klusterlet-config\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.392663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmlk4\" (UniqueName: \"kubernetes.io/projected/07f59c73-3208-4518-98d8-2794c26dca31-kube-api-access-kmlk4\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.392663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-ca\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.392663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b8590c5-8f6f-4294-850e-3ef461cf23c2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c6999d6c8-297dx\" (UID: \"5b8590c5-8f6f-4294-850e-3ef461cf23c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:50.392663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.392663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f59c73-3208-4518-98d8-2794c26dca31-tmp\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.392972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb6g7\" (UniqueName: \"kubernetes.io/projected/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-kube-api-access-qb6g7\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.392972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.392972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-hub\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.392972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2tr2\" (UniqueName: \"kubernetes.io/projected/5b8590c5-8f6f-4294-850e-3ef461cf23c2-kube-api-access-q2tr2\") pod \"managed-serviceaccount-addon-agent-6c6999d6c8-297dx\" (UID: \"5b8590c5-8f6f-4294-850e-3ef461cf23c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:50.392972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392780 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/55a38cc2-ffaa-46be-a929-a3d7fb71250d-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-4tkkk\" (UID: \"55a38cc2-ffaa-46be-a929-a3d7fb71250d\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.392972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.392799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.393768 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.393352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07f59c73-3208-4518-98d8-2794c26dca31-tmp\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.394110 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.394083 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-additional-service-ca\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.394242 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.394157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/55a38cc2-ffaa-46be-a929-a3d7fb71250d-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-4tkkk\" (UID: \"55a38cc2-ffaa-46be-a929-a3d7fb71250d\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.395748 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.395707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/55a38cc2-ffaa-46be-a929-a3d7fb71250d-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-4tkkk\" (UID: \"55a38cc2-ffaa-46be-a929-a3d7fb71250d\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.395748 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.395738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.395873 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.395777 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-ca\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.395926 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.395889 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.396393 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.396376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/07f59c73-3208-4518-98d8-2794c26dca31-klusterlet-config\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.396651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.396633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-hub\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.409100 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.409082 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb6g7\" (UniqueName: \"kubernetes.io/projected/2c1e33d3-e7ca-4306-8d56-829a5d79e4ae-kube-api-access-qb6g7\") pod \"cluster-proxy-proxy-agent-ccf66ccb5-c6gmj\" (UID: \"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.411713 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.411696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmlk4\" (UniqueName: \"kubernetes.io/projected/07f59c73-3208-4518-98d8-2794c26dca31-kube-api-access-kmlk4\") pod \"klusterlet-addon-workmgr-d856d94c6-86dbd\" (UID: \"07f59c73-3208-4518-98d8-2794c26dca31\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.436816 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.436794 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59fcd9fd6d-l6gbg"] Aug 04 15:11:50.437742 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:50.437721 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc330d986_f440_4bb4_b152_bdb21af2925e.slice/crio-078b1a40df17a51656e8bfbfc2be7c799c24674c6c3742bd2527793c2b271558 WatchSource:0}: Error finding container 078b1a40df17a51656e8bfbfc2be7c799c24674c6c3742bd2527793c2b271558: Status 404 returned error can't find the container with id 078b1a40df17a51656e8bfbfc2be7c799c24674c6c3742bd2527793c2b271558 Aug 04 15:11:50.488390 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.488359 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:50.493419 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.493370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b8590c5-8f6f-4294-850e-3ef461cf23c2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c6999d6c8-297dx\" (UID: \"5b8590c5-8f6f-4294-850e-3ef461cf23c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:50.493482 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.493422 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2tr2\" (UniqueName: \"kubernetes.io/projected/5b8590c5-8f6f-4294-850e-3ef461cf23c2-kube-api-access-q2tr2\") pod \"managed-serviceaccount-addon-agent-6c6999d6c8-297dx\" (UID: \"5b8590c5-8f6f-4294-850e-3ef461cf23c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:50.511763 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.511738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2tr2\" (UniqueName: \"kubernetes.io/projected/5b8590c5-8f6f-4294-850e-3ef461cf23c2-kube-api-access-q2tr2\") pod \"managed-serviceaccount-addon-agent-6c6999d6c8-297dx\" (UID: \"5b8590c5-8f6f-4294-850e-3ef461cf23c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:50.522777 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.522752 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" Aug 04 15:11:50.536973 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.536951 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" Aug 04 15:11:50.546449 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.546423 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:50.645867 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.645684 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:50.689923 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.689096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" event={"ID":"c330d986-f440-4bb4-b152-bdb21af2925e","Type":"ContainerStarted","Data":"54c1ab1461bda6b30436b381e21f0ddb8801995af5c052206e0bb8c3f6f943e5"} Aug 04 15:11:50.689923 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.689177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" event={"ID":"c330d986-f440-4bb4-b152-bdb21af2925e","Type":"ContainerStarted","Data":"078b1a40df17a51656e8bfbfc2be7c799c24674c6c3742bd2527793c2b271558"} Aug 04 15:11:50.691529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.691486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" event={"ID":"f2fb7ca7-9668-4869-a3df-5022e5c62cf1","Type":"ContainerStarted","Data":"e0a80ae975f51acce19f14ea79308d176b9fd3360c5e63c5c840fb86b035e2d2"} Aug 04 15:11:50.704265 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.704243 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk"] Aug 04 15:11:50.706749 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:50.706724 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a38cc2_ffaa_46be_a929_a3d7fb71250d.slice/crio-806eee573a5f467a0e67ba4c3173e2c44685020885a8f50ec8b2e00f206f9452 WatchSource:0}: Error finding container 806eee573a5f467a0e67ba4c3173e2c44685020885a8f50ec8b2e00f206f9452: Status 404 returned error can't find the container with id 806eee573a5f467a0e67ba4c3173e2c44685020885a8f50ec8b2e00f206f9452 Aug 04 15:11:50.755404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.755340 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd"] Aug 04 15:11:50.758415 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:50.758391 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f59c73_3208_4518_98d8_2794c26dca31.slice/crio-cc3b10e79d8b1b32eeca8d02fd676bfe1e2b40da09362c91b515738c498a7d21 WatchSource:0}: Error finding container cc3b10e79d8b1b32eeca8d02fd676bfe1e2b40da09362c91b515738c498a7d21: Status 404 returned error can't find the container with id cc3b10e79d8b1b32eeca8d02fd676bfe1e2b40da09362c91b515738c498a7d21 Aug 04 15:11:50.766230 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.766204 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj"] Aug 04 15:11:50.766758 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:50.766732 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c1e33d3_e7ca_4306_8d56_829a5d79e4ae.slice/crio-25ce986efb4d1c0265c6f3e7910f055b2355cef977dedf6d1d8088b02660ffd1 WatchSource:0}: Error finding container 25ce986efb4d1c0265c6f3e7910f055b2355cef977dedf6d1d8088b02660ffd1: Status 404 returned error can't find the container with id 25ce986efb4d1c0265c6f3e7910f055b2355cef977dedf6d1d8088b02660ffd1 Aug 04 15:11:50.771198 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.771158 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" podStartSLOduration=32.771145063 podStartE2EDuration="32.771145063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:50.770213026 +0000 UTC m=+76.124300785" watchObservedRunningTime="2026-08-04 15:11:50.771145063 +0000 UTC m=+76.125232877" Aug 04 15:11:50.830933 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:50.830897 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:51.114733 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.114655 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cwhcp\"" Aug 04 15:11:51.289186 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.289154 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:51.289370 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.289251 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:51.400272 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.400194 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:51.494407 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:51.494375 2559 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/managed-serviceaccount-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition Aug 04 15:11:51.494566 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:11:51.494459 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b8590c5-8f6f-4294-850e-3ef461cf23c2-hub-kubeconfig podName:5b8590c5-8f6f-4294-850e-3ef461cf23c2 nodeName:}" failed. No retries permitted until 2026-08-04 15:11:51.994442795 +0000 UTC m=+77.348530532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub-kubeconfig" (UniqueName: "kubernetes.io/secret/5b8590c5-8f6f-4294-850e-3ef461cf23c2-hub-kubeconfig") pod "managed-serviceaccount-addon-agent-6c6999d6c8-297dx" (UID: "5b8590c5-8f6f-4294-850e-3ef461cf23c2") : failed to sync secret cache: timed out waiting for the condition Aug 04 15:11:51.552220 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.552175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:51.628126 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.628055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:51.704853 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.704815 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" event={"ID":"55a38cc2-ffaa-46be-a929-a3d7fb71250d","Type":"ContainerStarted","Data":"806eee573a5f467a0e67ba4c3173e2c44685020885a8f50ec8b2e00f206f9452"} Aug 04 15:11:51.707742 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.707710 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" event={"ID":"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae","Type":"ContainerStarted","Data":"25ce986efb4d1c0265c6f3e7910f055b2355cef977dedf6d1d8088b02660ffd1"} Aug 04 15:11:51.712252 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.712004 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" event={"ID":"07f59c73-3208-4518-98d8-2794c26dca31","Type":"ContainerStarted","Data":"cc3b10e79d8b1b32eeca8d02fd676bfe1e2b40da09362c91b515738c498a7d21"} Aug 04 15:11:51.712443 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.712397 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:51.714231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.714078 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59fcd9fd6d-l6gbg" Aug 04 15:11:51.715867 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:51.715840 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Aug 04 15:11:52.004365 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:52.004275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b8590c5-8f6f-4294-850e-3ef461cf23c2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c6999d6c8-297dx\" (UID: \"5b8590c5-8f6f-4294-850e-3ef461cf23c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:52.008192 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:52.008162 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b8590c5-8f6f-4294-850e-3ef461cf23c2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c6999d6c8-297dx\" (UID: \"5b8590c5-8f6f-4294-850e-3ef461cf23c2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:52.053127 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:52.053093 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" Aug 04 15:11:52.796586 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:52.796525 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:54.663127 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:54.663105 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2bng9" Aug 04 15:11:57.051661 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.051625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:57.194468 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.194441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:57.211545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.211511 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx"] Aug 04 15:11:57.218075 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:11:57.218045 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b8590c5_8f6f_4294_850e_3ef461cf23c2.slice/crio-223be489cb57ee8e256feb44b03709273f5c808d3661b7c76222cba8ee258e2b WatchSource:0}: Error finding container 223be489cb57ee8e256feb44b03709273f5c808d3661b7c76222cba8ee258e2b: Status 404 returned error can't find the container with id 223be489cb57ee8e256feb44b03709273f5c808d3661b7c76222cba8ee258e2b Aug 04 15:11:57.714399 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.714357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:57.741485 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.741391 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" event={"ID":"55a38cc2-ffaa-46be-a929-a3d7fb71250d","Type":"ContainerStarted","Data":"3e95ec40db9e8e322014463d7e36f472d38986f4673a95666c5f6b975e1c3101"} Aug 04 15:11:57.743085 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.743054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" event={"ID":"f2fb7ca7-9668-4869-a3df-5022e5c62cf1","Type":"ContainerStarted","Data":"1c0e1a5d8fafbfb14e0228b90407082b95b4b715ebb5dc9566a7546ddc42a098"} Aug 04 15:11:57.744170 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.744146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" event={"ID":"5b8590c5-8f6f-4294-850e-3ef461cf23c2","Type":"ContainerStarted","Data":"223be489cb57ee8e256feb44b03709273f5c808d3661b7c76222cba8ee258e2b"} Aug 04 15:11:57.746347 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.746319 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" event={"ID":"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae","Type":"ContainerStarted","Data":"ef5323e668e90aa5989ff1f81bb5ad70eca4444f3a78aab807ab55074927649b"} Aug 04 15:11:57.746471 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.746354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" event={"ID":"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae","Type":"ContainerStarted","Data":"bde753740abc276ec1b6f40ed8389a5421fe2d484dec12480b0319213fd68f73"} Aug 04 15:11:57.746471 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.746373 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" event={"ID":"2c1e33d3-e7ca-4306-8d56-829a5d79e4ae","Type":"ContainerStarted","Data":"f02047a6bef2f81ec1db28b02630804157970977fa8a2083f25d7a98f647f74f"} Aug 04 15:11:57.747831 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.747812 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" event={"ID":"07f59c73-3208-4518-98d8-2794c26dca31","Type":"ContainerStarted","Data":"1f9b2b2b83bf2b85f89098e59625e981a638ce6abca1dbcd1e58aafb7d06da75"} Aug 04 15:11:57.748036 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.748016 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:57.749751 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.749732 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" Aug 04 15:11:57.767122 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.767082 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-4tkkk" podStartSLOduration=1.294776489 podStartE2EDuration="7.767070063s" totalImagesPullingTime="6.472293574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:57.765600183 +0000 UTC m=+83.119687938" watchObservedRunningTime="2026-08-04 15:11:57.767070063 +0000 UTC m=+83.121157822" Aug 04 15:11:57.818562 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.818506 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-ccf66ccb5-c6gmj" podStartSLOduration=1.3620494490000001 podStartE2EDuration="7.818484171s" totalImagesPullingTime="6.456434722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:57.818460856 +0000 UTC m=+83.172548657" watchObservedRunningTime="2026-08-04 15:11:57.818484171 +0000 UTC m=+83.172571930" Aug 04 15:11:57.879412 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.879347 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-sxq7z" podStartSLOduration=34.219951652 podStartE2EDuration="40.879331465s" totalImagesPullingTime="6.659379813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:57.877890142 +0000 UTC m=+83.231977902" watchObservedRunningTime="2026-08-04 15:11:57.879331465 +0000 UTC m=+83.233419283" Aug 04 15:11:57.880777 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:57.880745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:58.029844 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:58.029754 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:58.120392 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:58.120351 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:58.340384 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:58.340160 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:11:58.438808 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:11:58.438760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:00.080648 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:00.080614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:00.179361 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:00.179328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:00.763348 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:00.763314 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" event={"ID":"5b8590c5-8f6f-4294-850e-3ef461cf23c2","Type":"ContainerStarted","Data":"a85aee580ddff049806e4ed3321fbd558f3c896b239f6b681bdd0ccebb2d6be4"} Aug 04 15:12:00.781930 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:00.781874 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d856d94c6-86dbd" podStartSLOduration=4.327871647 podStartE2EDuration="10.78185696s" totalImagesPullingTime="6.453985313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:11:57.912262023 +0000 UTC m=+83.266349773" watchObservedRunningTime="2026-08-04 15:12:00.78185696 +0000 UTC m=+86.135944718" Aug 04 15:12:02.118513 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:02.118478 2559 patch_prober.go:28] interesting pod/image-registry-7bb959c8b7-dv5dw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Aug 04 15:12:02.118896 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:02.118533 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" podUID="3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:12:03.245258 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.245224 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:03.282479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.282451 2559 scope.go:117] "RemoveContainer" containerID="62f8075d0e02d4c1b4877efa25868ee54d8edf57b2dd805bc6f7af03b95738aa" Aug 04 15:12:03.346959 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.346892 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:03.654790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.654700 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:12:03.689255 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.689206 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c6999d6c8-297dx" podStartSLOduration=10.943738893999999 podStartE2EDuration="13.68918953s" totalImagesPullingTime="2.745450636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:12:00.780778342 +0000 UTC m=+86.134866101" watchObservedRunningTime="2026-08-04 15:12:03.68918953 +0000 UTC m=+89.043277282" Aug 04 15:12:03.778226 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.778199 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:12:03.778375 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.778291 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" event={"ID":"814798a3-c8db-4727-920f-dfca65127d18","Type":"ContainerStarted","Data":"0a51d4e93d7f212339ce052633b07e08b4d822dc2b9841892c1cf30d4cd4b350"} Aug 04 15:12:03.778571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.778552 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:12:03.799179 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.799128 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" podStartSLOduration=40.188892895 podStartE2EDuration="45.799113377s" totalImagesPullingTime="5.610220482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:11:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:12:03.79827674 +0000 UTC m=+89.152364502" watchObservedRunningTime="2026-08-04 15:12:03.799113377 +0000 UTC m=+89.153201135" Aug 04 15:12:03.850760 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:03.850727 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-9j44d" Aug 04 15:12:05.133208 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:05.133172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:05.233223 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:05.233168 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:05.577920 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:05.577884 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:05.740094 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:05.740060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:06.034321 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:06.033495 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:06.437835 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:06.437745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:06.574686 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:06.574652 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:06.793746 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:06.793715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:06.926197 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:06.926157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:07.026425 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:07.026387 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:07.578951 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:07.578921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:07.657858 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:07.657822 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:08.388556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.388517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:08.535206 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.535175 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:08.669631 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.669540 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pwn2f"] Aug 04 15:12:08.672725 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.672705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.675748 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.675727 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Aug 04 15:12:08.675878 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.675769 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Aug 04 15:12:08.675878 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.675833 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qbrvb\"" Aug 04 15:12:08.676056 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.676040 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Aug 04 15:12:08.676743 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.676727 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Aug 04 15:12:08.717023 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.716982 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:08.742810 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.742781 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-sys\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.742982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.742816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.742982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.742837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-wtmp\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.742982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.742862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-textfile\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.742982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.742890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-tls\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.742982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.742951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.743176 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.742985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3764a934-28c3-453f-90f2-d2e3fb7ea247-metrics-client-ca\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.743176 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.743002 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgv7\" (UniqueName: \"kubernetes.io/projected/3764a934-28c3-453f-90f2-d2e3fb7ea247-kube-api-access-grgv7\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.743176 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.743020 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-root\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.803854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.803821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:08.843603 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-sys\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.843770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.843770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843650 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-wtmp\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.843770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-sys\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.843770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-textfile\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.843770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-tls\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.843770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3764a934-28c3-453f-90f2-d2e3fb7ea247-metrics-client-ca\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844065 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:12:08.843872 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Aug 04 15:12:08.844065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-wtmp\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844065 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:12:08.843956 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-tls podName:3764a934-28c3-453f-90f2-d2e3fb7ea247 nodeName:}" failed. No retries permitted until 2026-08-04 15:12:09.343934635 +0000 UTC m=+94.698022384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-tls") pod "node-exporter-pwn2f" (UID: "3764a934-28c3-453f-90f2-d2e3fb7ea247") : secret "node-exporter-tls" not found Aug 04 15:12:08.844065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.843983 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgv7\" (UniqueName: \"kubernetes.io/projected/3764a934-28c3-453f-90f2-d2e3fb7ea247-kube-api-access-grgv7\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.844023 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-root\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844398 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.844102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3764a934-28c3-453f-90f2-d2e3fb7ea247-root\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844398 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.844138 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-textfile\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844481 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.844402 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3764a934-28c3-453f-90f2-d2e3fb7ea247-metrics-client-ca\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.844515 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.844487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.846216 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.846188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:08.859381 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:08.859355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgv7\" (UniqueName: \"kubernetes.io/projected/3764a934-28c3-453f-90f2-d2e3fb7ea247-kube-api-access-grgv7\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:09.163521 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:09.163484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:09.310927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:09.310886 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:09.347932 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:09.347892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-tls\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:09.350294 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:09.350266 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3764a934-28c3-453f-90f2-d2e3fb7ea247-node-exporter-tls\") pod \"node-exporter-pwn2f\" (UID: \"3764a934-28c3-453f-90f2-d2e3fb7ea247\") " pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:09.583000 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:09.582963 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwn2f" Aug 04 15:12:09.593886 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:12:09.593857 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3764a934_28c3_453f_90f2_d2e3fb7ea247.slice/crio-fedfe07750fbb62b749dadf382848cfdfbf45753a00fd686c1fe7331b390b3a2 WatchSource:0}: Error finding container fedfe07750fbb62b749dadf382848cfdfbf45753a00fd686c1fe7331b390b3a2: Status 404 returned error can't find the container with id fedfe07750fbb62b749dadf382848cfdfbf45753a00fd686c1fe7331b390b3a2 Aug 04 15:12:09.807093 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:09.807057 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwn2f" event={"ID":"3764a934-28c3-453f-90f2-d2e3fb7ea247","Type":"ContainerStarted","Data":"fedfe07750fbb62b749dadf382848cfdfbf45753a00fd686c1fe7331b390b3a2"} Aug 04 15:12:10.686778 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:10.686739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:10.784479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:10.784438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:11.711240 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:11.711132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:11.828804 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:11.828750 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:12.295791 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:12.295704 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:12.483280 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:12.483247 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:12.643194 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:12.643121 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kl72z" Aug 04 15:12:12.656322 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:12.656283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:12.822436 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:12.822395 2559 generic.go:356] "Generic (PLEG): container finished" podID="3764a934-28c3-453f-90f2-d2e3fb7ea247" containerID="ff550f9ecee6bf219ae61267345ba93731b10279073c8a87438e03a8e5b2ede4" exitCode=0 Aug 04 15:12:12.822884 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:12.822486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwn2f" event={"ID":"3764a934-28c3-453f-90f2-d2e3fb7ea247","Type":"ContainerDied","Data":"ff550f9ecee6bf219ae61267345ba93731b10279073c8a87438e03a8e5b2ede4"} Aug 04 15:12:13.261047 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:13.261014 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bb959c8b7-dv5dw"] Aug 04 15:12:13.393945 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:13.393910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:13.501885 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:13.501852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:13.829084 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:13.829051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwn2f" event={"ID":"3764a934-28c3-453f-90f2-d2e3fb7ea247","Type":"ContainerStarted","Data":"00156fcbff32ea694ae8ef8d8fddd48890a77aef773642bdbe8b3d8d72244961"} Aug 04 15:12:13.829572 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:13.829106 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwn2f" event={"ID":"3764a934-28c3-453f-90f2-d2e3fb7ea247","Type":"ContainerStarted","Data":"47a7c31e71460b23acdb68f48f7ad9f1f116ad518c60141dc5676a8ab0fab622"} Aug 04 15:12:13.849759 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:13.849712 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pwn2f" podStartSLOduration=3.404709366 podStartE2EDuration="5.849698087s" totalImagesPullingTime="2.37983124s" totalInitContainerRuntime="65.157481ms" isStatefulPod=true podCreationTimestamp="2026-08-04 15:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:12:13.848709692 +0000 UTC m=+99.202797451" watchObservedRunningTime="2026-08-04 15:12:13.849698087 +0000 UTC m=+99.203785874" Aug 04 15:12:14.188177 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:14.188102 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:14.452639 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:14.452601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:14.553046 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:14.553014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:15.068082 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:15.068052 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:15.296771 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:15.296734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:18.731504 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:18.731470 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:18.939880 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:18.939837 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:19.030269 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:19.030172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:19.167978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:19.167946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:19.243759 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:19.243731 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:21.201920 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:21.201888 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:21.278856 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:21.278827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:23.403909 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:23.403877 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:23.492066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:23.492024 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:24.426476 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:24.426439 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:24.515835 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:24.515795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:26.606709 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:26.606671 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:26.713782 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:26.713745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:28.790222 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:28.790192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:29.078235 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:29.078145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:31.180443 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:31.180407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:31.267261 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:31.267228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:38.281011 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.280951 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" podUID="3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" containerName="registry" containerID="cri-o://392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782" gracePeriod=30 Aug 04 15:12:38.530045 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.530018 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:12:38.588988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.588913 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2czll\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-kube-api-access-2czll\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.588988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.588965 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-ca-trust-extracted\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.589225 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589034 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-installation-pull-secrets\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.589225 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589058 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-image-registry-private-configuration\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.589225 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589085 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-bound-sa-token\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.589225 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589170 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-certificates\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.589430 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589223 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.589430 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589266 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-trusted-ca\") pod \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\" (UID: \"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931\") " Aug 04 15:12:38.589768 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589615 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:12:38.589922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.589895 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:12:38.591668 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.591632 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:12:38.591772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.591745 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:12:38.591837 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.591796 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:12:38.591882 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.591850 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:12:38.591946 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.591925 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-kube-api-access-2czll" (OuterVolumeSpecName: "kube-api-access-2czll") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "kube-api-access-2czll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:12:38.597209 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.597178 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" (UID: "3c793f63-8a0d-40e5-8f6a-61b6ce8a7931"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 15:12:38.690773 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690738 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-trusted-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.690773 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690765 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2czll\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-kube-api-access-2czll\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.690773 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690775 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-ca-trust-extracted\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.690773 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690783 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-installation-pull-secrets\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.691014 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690792 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-image-registry-private-configuration\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.691014 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690801 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-bound-sa-token\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.691014 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690809 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-certificates\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.691014 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.690825 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931-registry-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:12:38.926391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.926295 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" containerID="392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782" exitCode=0 Aug 04 15:12:38.926391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.926346 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" Aug 04 15:12:38.926610 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.926347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" event={"ID":"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931","Type":"ContainerDied","Data":"392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782"} Aug 04 15:12:38.926610 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.926470 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bb959c8b7-dv5dw" event={"ID":"3c793f63-8a0d-40e5-8f6a-61b6ce8a7931","Type":"ContainerDied","Data":"12f5d6b388d9db9574778427f66773c2c77c37650d8eb437de922abe805878db"} Aug 04 15:12:38.926610 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.926494 2559 scope.go:117] "RemoveContainer" containerID="392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782" Aug 04 15:12:38.934946 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.934926 2559 scope.go:117] "RemoveContainer" containerID="392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782" Aug 04 15:12:38.935223 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:12:38.935205 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782\": container with ID starting with 392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782 not found: ID does not exist" containerID="392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782" Aug 04 15:12:38.935268 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.935232 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782"} err="failed to get container status \"392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782\": rpc error: code = NotFound desc = could not find container \"392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782\": container with ID starting with 392a34e7d0204f5243273d10bfc0ad1af73df0c6289b3454010c2d3ba57a5782 not found: ID does not exist" Aug 04 15:12:38.950758 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.950736 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bb959c8b7-dv5dw"] Aug 04 15:12:38.958850 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:38.958827 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bb959c8b7-dv5dw"] Aug 04 15:12:39.287169 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:39.287126 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" path="/var/lib/kubelet/pods/3c793f63-8a0d-40e5-8f6a-61b6ce8a7931/volumes" Aug 04 15:12:39.388260 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:39.388226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:39.473622 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:39.473563 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:40.568997 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:40.568954 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:40.936435 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:40.936349 2559 generic.go:356] "Generic (PLEG): container finished" podID="f257daa1-126f-4549-a16f-4a5967506834" containerID="e42938210aec6c2b4d0fe9160d242fd6eeca10bf0cf8d905c05d1557e3af78f7" exitCode=0 Aug 04 15:12:40.936620 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:40.936425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" event={"ID":"f257daa1-126f-4549-a16f-4a5967506834","Type":"ContainerDied","Data":"e42938210aec6c2b4d0fe9160d242fd6eeca10bf0cf8d905c05d1557e3af78f7"} Aug 04 15:12:40.936823 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:40.936806 2559 scope.go:117] "RemoveContainer" containerID="e42938210aec6c2b4d0fe9160d242fd6eeca10bf0cf8d905c05d1557e3af78f7" Aug 04 15:12:41.942966 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:41.942926 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-zkbnp" event={"ID":"f257daa1-126f-4549-a16f-4a5967506834","Type":"ContainerStarted","Data":"b06698a0a75ef318445a6f853299e163799d196bd92d73c314672aa175c09ab0"} Aug 04 15:12:42.178968 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:42.178919 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:42.281448 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:42.281366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:44.956062 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:44.956031 2559 generic.go:356] "Generic (PLEG): container finished" podID="e8b23c6f-dabe-4c2d-9993-e7682fdb4c26" containerID="5c6221a82b8d18d1bf54169fd1de40bc3bd2bc8ab9468b01d71562b186c2af14" exitCode=0 Aug 04 15:12:44.956419 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:44.956092 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-z5kg4" event={"ID":"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26","Type":"ContainerDied","Data":"5c6221a82b8d18d1bf54169fd1de40bc3bd2bc8ab9468b01d71562b186c2af14"} Aug 04 15:12:44.956419 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:44.956406 2559 scope.go:117] "RemoveContainer" containerID="5c6221a82b8d18d1bf54169fd1de40bc3bd2bc8ab9468b01d71562b186c2af14" Aug 04 15:12:45.961309 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:45.961276 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-z5kg4" event={"ID":"e8b23c6f-dabe-4c2d-9993-e7682fdb4c26","Type":"ContainerStarted","Data":"6c5b87119de204d00c92b877c4b043cb30110ce9ffcbd2e3441a2d94cff417a0"} Aug 04 15:12:50.983148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:50.983114 2559 generic.go:356] "Generic (PLEG): container finished" podID="30c90e46-5199-4d39-8858-910139a1cbcf" containerID="d2a8392049d841c6446ccd7231dbf1dba209cfb9be4632dea522ea28667e6b9b" exitCode=0 Aug 04 15:12:50.983655 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:50.983177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" event={"ID":"30c90e46-5199-4d39-8858-910139a1cbcf","Type":"ContainerDied","Data":"d2a8392049d841c6446ccd7231dbf1dba209cfb9be4632dea522ea28667e6b9b"} Aug 04 15:12:50.983655 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:50.983572 2559 scope.go:117] "RemoveContainer" containerID="d2a8392049d841c6446ccd7231dbf1dba209cfb9be4632dea522ea28667e6b9b" Aug 04 15:12:51.988731 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:51.988696 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-ds9bz" event={"ID":"30c90e46-5199-4d39-8858-910139a1cbcf","Type":"ContainerStarted","Data":"2e0079e1c5022e0c18d8864a28ad26f2b5e99976bcf7fc01661647ab754ccccb"} Aug 04 15:12:54.412791 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:54.412750 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:12:54.542705 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:12:54.542673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:10.595618 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:10.595560 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:19.594306 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:19.594276 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:19.690206 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:19.690167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:28.129497 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:28.129412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:28.704286 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:28.704247 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:38.933601 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:38.933537 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:39.051670 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:39.051632 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:49.840984 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:49.840940 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:49.942342 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:49.942306 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:13:50.130001 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:13:50.129912 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:14.721508 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:14.721466 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:16.534438 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:16.534396 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:16.658407 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:16.658373 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:43.505265 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:43.505229 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:43.590420 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:43.590387 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:50.750733 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:50.750692 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:55.532504 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:55.532466 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:14:55.645043 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:14:55.645001 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:02.170891 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:02.170803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:02.338750 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:02.338718 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:09.385179 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:09.385140 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:09.484251 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:09.484195 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:09.553712 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:09.553680 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:09.735134 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:09.735095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:35.055982 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:35.055953 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:15:35.059372 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:35.059341 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:15:35.061338 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:35.061317 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:15:35.064977 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:35.064953 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:15:35.069268 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:35.069250 2559 kubelet.go:1648] "Image garbage collection succeeded" Aug 04 15:15:55.386128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:55.386089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:15:56.579716 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:15:56.579674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:22.548889 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:22.548850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:23.893082 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:23.893046 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:26.357446 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:26.357411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:26.458463 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:26.458431 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:27.670242 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:27.670165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:27.785673 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:27.785631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:28.477032 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:28.476989 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:28.589883 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:28.589847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:30.886437 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:30.886405 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:31.062638 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:31.062598 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:33.149757 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:33.149718 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:33.250595 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:33.250553 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:38.469484 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:38.469409 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:44.761067 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:44.761030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:44.861991 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:44.861957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:50.036027 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:50.035984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:50.135624 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:50.135563 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:51.772493 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:51.772460 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:55.561276 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:55.561245 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:16:55.618792 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:16:55.618768 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:04.175842 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:04.175805 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:04.267667 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:04.267631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:04.576173 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:04.576139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:04.674477 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:04.674438 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:05.177510 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.177473 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:05.586562 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.586520 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz"] Aug 04 15:17:05.586856 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.586841 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" containerName="registry" Aug 04 15:17:05.586911 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.586858 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" containerName="registry" Aug 04 15:17:05.586946 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.586915 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c793f63-8a0d-40e5-8f6a-61b6ce8a7931" containerName="registry" Aug 04 15:17:05.589831 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.589815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.594322 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.594300 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Aug 04 15:17:05.594439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.594323 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Aug 04 15:17:05.594439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.594367 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fjn9c\"" Aug 04 15:17:05.595373 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.595354 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Aug 04 15:17:05.595455 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.595403 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Aug 04 15:17:05.604019 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.603993 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz"] Aug 04 15:17:05.666007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.665971 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrbr\" (UniqueName: \"kubernetes.io/projected/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-kube-api-access-4vrbr\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.666170 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.666023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.666170 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.666094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-tmp\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.666170 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.666143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-manifests\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.666170 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.666169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-webhook-cert\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.767145 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.767110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.767328 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.767152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-tmp\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.767328 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.767175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-manifests\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.767328 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.767198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-webhook-cert\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.767328 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.767219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrbr\" (UniqueName: \"kubernetes.io/projected/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-kube-api-access-4vrbr\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.767668 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.767645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-manifests\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.769389 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.769362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-tmp\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.769671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.769655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-webhook-cert\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.769723 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.769655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.774921 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.774899 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrbr\" (UniqueName: \"kubernetes.io/projected/f2d73a7a-cd9f-460a-b470-4ea1a3f66292-kube-api-access-4vrbr\") pod \"opendatahub-operator-controller-manager-77c9c696b6-dhhvz\" (UID: \"f2d73a7a-cd9f-460a-b470-4ea1a3f66292\") " pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:05.900102 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:05.900015 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:06.035819 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:06.035794 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz"] Aug 04 15:17:06.038256 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:17:06.038225 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d73a7a_cd9f_460a_b470_4ea1a3f66292.slice/crio-052f0cc036f2a9e889567607f79f329bcb9552a84695dc34d3fcc81ae143d8c1 WatchSource:0}: Error finding container 052f0cc036f2a9e889567607f79f329bcb9552a84695dc34d3fcc81ae143d8c1: Status 404 returned error can't find the container with id 052f0cc036f2a9e889567607f79f329bcb9552a84695dc34d3fcc81ae143d8c1 Aug 04 15:17:06.040603 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:06.040568 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:17:06.953564 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:06.953520 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" event={"ID":"f2d73a7a-cd9f-460a-b470-4ea1a3f66292","Type":"ContainerStarted","Data":"052f0cc036f2a9e889567607f79f329bcb9552a84695dc34d3fcc81ae143d8c1"} Aug 04 15:17:07.116589 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:07.116543 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:07.230352 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:07.230248 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:09.966891 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:09.966854 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" event={"ID":"f2d73a7a-cd9f-460a-b470-4ea1a3f66292","Type":"ContainerStarted","Data":"cbeaf30381ec889478609780f319734f8b663ef94fd6917c376f6b2b010ceb95"} Aug 04 15:17:10.972510 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:10.972470 2559 generic.go:356] "Generic (PLEG): container finished" podID="f2d73a7a-cd9f-460a-b470-4ea1a3f66292" containerID="cbeaf30381ec889478609780f319734f8b663ef94fd6917c376f6b2b010ceb95" exitCode=0 Aug 04 15:17:10.972977 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:10.972556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" event={"ID":"f2d73a7a-cd9f-460a-b470-4ea1a3f66292","Type":"ContainerDied","Data":"cbeaf30381ec889478609780f319734f8b663ef94fd6917c376f6b2b010ceb95"} Aug 04 15:17:15.993355 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:15.993313 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" event={"ID":"f2d73a7a-cd9f-460a-b470-4ea1a3f66292","Type":"ContainerStarted","Data":"ec70f2d1e858cbb29411f99c6211e10476b27dae545c9c4a2678e40a593dab4b"} Aug 04 15:17:15.993742 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:15.993400 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:16.027262 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:16.027207 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" podStartSLOduration=2.434690812 podStartE2EDuration="11.027190986s" totalImagesPullingTime="7.505593589s" totalInitContainerRuntime="1.086906585s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:17:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:17:16.024807381 +0000 UTC m=+401.378895192" watchObservedRunningTime="2026-08-04 15:17:16.027190986 +0000 UTC m=+401.381278748" Aug 04 15:17:16.757553 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:16.757517 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:16.857688 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:16.857658 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:23.514531 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:23.514499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:26.998963 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:26.998929 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77c9c696b6-dhhvz" Aug 04 15:17:28.432529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.432491 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:28.525972 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.525933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:28.742073 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.742039 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:28.910927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.910893 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9dff769-mctsj"] Aug 04 15:17:28.913840 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.913825 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:28.918631 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.918611 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-29mzv\"" Aug 04 15:17:28.919535 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.919514 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Aug 04 15:17:28.924800 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.924782 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Aug 04 15:17:28.941643 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.941614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:28.944645 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:28.944621 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9dff769-mctsj"] Aug 04 15:17:29.065988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.065906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309833bf-fc35-445a-9c48-c31ccd4c1f2d-tls-certs\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.065988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.065938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55cbr\" (UniqueName: \"kubernetes.io/projected/309833bf-fc35-445a-9c48-c31ccd4c1f2d-kube-api-access-55cbr\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.065988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.065963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309833bf-fc35-445a-9c48-c31ccd4c1f2d-tmp\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.166974 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.166934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309833bf-fc35-445a-9c48-c31ccd4c1f2d-tls-certs\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.167133 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.166977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55cbr\" (UniqueName: \"kubernetes.io/projected/309833bf-fc35-445a-9c48-c31ccd4c1f2d-kube-api-access-55cbr\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.167133 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.167010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309833bf-fc35-445a-9c48-c31ccd4c1f2d-tmp\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.169195 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.169171 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/309833bf-fc35-445a-9c48-c31ccd4c1f2d-tmp\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.169314 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.169300 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309833bf-fc35-445a-9c48-c31ccd4c1f2d-tls-certs\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.184253 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.184225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55cbr\" (UniqueName: \"kubernetes.io/projected/309833bf-fc35-445a-9c48-c31ccd4c1f2d-kube-api-access-55cbr\") pod \"kube-auth-proxy-86b9dff769-mctsj\" (UID: \"309833bf-fc35-445a-9c48-c31ccd4c1f2d\") " pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.222839 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.222799 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" Aug 04 15:17:29.238286 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.238262 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jtsss"] Aug 04 15:17:29.244655 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.243415 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:29.248404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.248368 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Aug 04 15:17:29.248549 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.248425 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-556rb\"" Aug 04 15:17:29.267412 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.267388 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jtsss"] Aug 04 15:17:29.333380 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.333347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:29.368818 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.368783 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwtz\" (UniqueName: \"kubernetes.io/projected/244b7eea-6747-4456-99d2-c82257b5bf6e-kube-api-access-gvwtz\") pod \"odh-model-controller-7998c67dc4-jtsss\" (UID: \"244b7eea-6747-4456-99d2-c82257b5bf6e\") " pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:29.368990 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.368839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/244b7eea-6747-4456-99d2-c82257b5bf6e-cert\") pod \"odh-model-controller-7998c67dc4-jtsss\" (UID: \"244b7eea-6747-4456-99d2-c82257b5bf6e\") " pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:29.388732 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.388664 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86b9dff769-mctsj"] Aug 04 15:17:29.391345 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:17:29.391315 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309833bf_fc35_445a_9c48_c31ccd4c1f2d.slice/crio-8e63fa4f2609ceaa7540ddb237960151a6c7a78edb68c06a894937bb90716221 WatchSource:0}: Error finding container 8e63fa4f2609ceaa7540ddb237960151a6c7a78edb68c06a894937bb90716221: Status 404 returned error can't find the container with id 8e63fa4f2609ceaa7540ddb237960151a6c7a78edb68c06a894937bb90716221 Aug 04 15:17:29.470113 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.470079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/244b7eea-6747-4456-99d2-c82257b5bf6e-cert\") pod \"odh-model-controller-7998c67dc4-jtsss\" (UID: \"244b7eea-6747-4456-99d2-c82257b5bf6e\") " pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:29.470539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.470228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwtz\" (UniqueName: \"kubernetes.io/projected/244b7eea-6747-4456-99d2-c82257b5bf6e-kube-api-access-gvwtz\") pod \"odh-model-controller-7998c67dc4-jtsss\" (UID: \"244b7eea-6747-4456-99d2-c82257b5bf6e\") " pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:29.470539 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:17:29.470253 2559 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Aug 04 15:17:29.470539 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:17:29.470331 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244b7eea-6747-4456-99d2-c82257b5bf6e-cert podName:244b7eea-6747-4456-99d2-c82257b5bf6e nodeName:}" failed. No retries permitted until 2026-08-04 15:17:29.970307669 +0000 UTC m=+415.324395420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/244b7eea-6747-4456-99d2-c82257b5bf6e-cert") pod "odh-model-controller-7998c67dc4-jtsss" (UID: "244b7eea-6747-4456-99d2-c82257b5bf6e") : secret "odh-model-controller-webhook-cert" not found Aug 04 15:17:29.478938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.478907 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:29.499447 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.499424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwtz\" (UniqueName: \"kubernetes.io/projected/244b7eea-6747-4456-99d2-c82257b5bf6e-kube-api-access-gvwtz\") pod \"odh-model-controller-7998c67dc4-jtsss\" (UID: \"244b7eea-6747-4456-99d2-c82257b5bf6e\") " pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:29.668232 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.668145 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:29.809693 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.809659 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:29.976078 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.976047 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/244b7eea-6747-4456-99d2-c82257b5bf6e-cert\") pod \"odh-model-controller-7998c67dc4-jtsss\" (UID: \"244b7eea-6747-4456-99d2-c82257b5bf6e\") " pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:29.978364 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:29.978339 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/244b7eea-6747-4456-99d2-c82257b5bf6e-cert\") pod \"odh-model-controller-7998c67dc4-jtsss\" (UID: \"244b7eea-6747-4456-99d2-c82257b5bf6e\") " pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:30.051075 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.051041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" event={"ID":"309833bf-fc35-445a-9c48-c31ccd4c1f2d","Type":"ContainerStarted","Data":"8e63fa4f2609ceaa7540ddb237960151a6c7a78edb68c06a894937bb90716221"} Aug 04 15:17:30.159281 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.159248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:30.313669 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.313642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-jtsss"] Aug 04 15:17:30.315450 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:17:30.315425 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244b7eea_6747_4456_99d2_c82257b5bf6e.slice/crio-2f213c6a02a0499d48e0538c9b1b56691bbcef04a9d442fcdcfd50e4e8c70d23 WatchSource:0}: Error finding container 2f213c6a02a0499d48e0538c9b1b56691bbcef04a9d442fcdcfd50e4e8c70d23: Status 404 returned error can't find the container with id 2f213c6a02a0499d48e0538c9b1b56691bbcef04a9d442fcdcfd50e4e8c70d23 Aug 04 15:17:30.632662 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.632558 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bd7pk"] Aug 04 15:17:30.636365 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.636340 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:30.642026 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.642003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Aug 04 15:17:30.663767 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.663732 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bd7pk"] Aug 04 15:17:30.781989 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.781928 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d286k\" (UniqueName: \"kubernetes.io/projected/1747d4d9-7474-45ae-8b9f-0889f79b22cd-kube-api-access-d286k\") pod \"model-serving-api-54d89bd5fc-bd7pk\" (UID: \"1747d4d9-7474-45ae-8b9f-0889f79b22cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:30.782180 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.782004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1747d4d9-7474-45ae-8b9f-0889f79b22cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-bd7pk\" (UID: \"1747d4d9-7474-45ae-8b9f-0889f79b22cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:30.883696 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.883455 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d286k\" (UniqueName: \"kubernetes.io/projected/1747d4d9-7474-45ae-8b9f-0889f79b22cd-kube-api-access-d286k\") pod \"model-serving-api-54d89bd5fc-bd7pk\" (UID: \"1747d4d9-7474-45ae-8b9f-0889f79b22cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:30.883696 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.883534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1747d4d9-7474-45ae-8b9f-0889f79b22cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-bd7pk\" (UID: \"1747d4d9-7474-45ae-8b9f-0889f79b22cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:30.888385 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.888328 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1747d4d9-7474-45ae-8b9f-0889f79b22cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-bd7pk\" (UID: \"1747d4d9-7474-45ae-8b9f-0889f79b22cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:30.900097 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.900030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d286k\" (UniqueName: \"kubernetes.io/projected/1747d4d9-7474-45ae-8b9f-0889f79b22cd-kube-api-access-d286k\") pod \"model-serving-api-54d89bd5fc-bd7pk\" (UID: \"1747d4d9-7474-45ae-8b9f-0889f79b22cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:30.951464 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:30.951284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:31.060680 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.060645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" event={"ID":"244b7eea-6747-4456-99d2-c82257b5bf6e","Type":"ContainerStarted","Data":"2f213c6a02a0499d48e0538c9b1b56691bbcef04a9d442fcdcfd50e4e8c70d23"} Aug 04 15:17:31.115033 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.114955 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-bd7pk"] Aug 04 15:17:31.118961 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:17:31.118926 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1747d4d9_7474_45ae_8b9f_0889f79b22cd.slice/crio-bcc55b1696681599e811ad5deae268414437e83dc494500cbcf06b658a7a2bfa WatchSource:0}: Error finding container bcc55b1696681599e811ad5deae268414437e83dc494500cbcf06b658a7a2bfa: Status 404 returned error can't find the container with id bcc55b1696681599e811ad5deae268414437e83dc494500cbcf06b658a7a2bfa Aug 04 15:17:31.149265 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.149191 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428"] Aug 04 15:17:31.154011 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.153989 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.156524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.156485 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Aug 04 15:17:31.156649 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.156590 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Aug 04 15:17:31.156966 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.156944 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Aug 04 15:17:31.157270 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.157250 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Aug 04 15:17:31.157411 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.157386 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8qn28\"" Aug 04 15:17:31.168853 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.168834 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428"] Aug 04 15:17:31.222280 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.222240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:31.286864 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.286831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.287053 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.286888 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.287053 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.286954 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9c6835-dff6-45b6-ac8e-120633dd429e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.287053 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.287013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.287213 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.287058 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.287213 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.287086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.287213 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.287189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87qd\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-kube-api-access-d87qd\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.302655 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.302402 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:31.388335 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.388296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87qd\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-kube-api-access-d87qd\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.388516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.388370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.388516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.388408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.388516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.388433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9c6835-dff6-45b6-ac8e-120633dd429e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.388516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.388463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.389183 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.388879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.389183 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.388958 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.389358 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.389254 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.391817 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.391792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.392152 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.392133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.392882 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.392857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.394004 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.393984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9c6835-dff6-45b6-ac8e-120633dd429e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.397471 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.397448 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.398052 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.398028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87qd\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-kube-api-access-d87qd\") pod \"istiod-openshift-gateway-7b7995d5cd-tz428\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:31.467162 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:31.467127 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:32.066898 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:32.066858 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" event={"ID":"1747d4d9-7474-45ae-8b9f-0889f79b22cd","Type":"ContainerStarted","Data":"bcc55b1696681599e811ad5deae268414437e83dc494500cbcf06b658a7a2bfa"} Aug 04 15:17:32.396551 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:32.396450 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:32.710468 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:32.710442 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428"] Aug 04 15:17:32.712241 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:17:32.712211 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9c6835_dff6_45b6_ac8e_120633dd429e.slice/crio-8e02cdb15cc1a280ad489bdb7b0d875917c6d6bc254a259be81495d7027af9ab WatchSource:0}: Error finding container 8e02cdb15cc1a280ad489bdb7b0d875917c6d6bc254a259be81495d7027af9ab: Status 404 returned error can't find the container with id 8e02cdb15cc1a280ad489bdb7b0d875917c6d6bc254a259be81495d7027af9ab Aug 04 15:17:33.074744 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:33.074671 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" event={"ID":"ac9c6835-dff6-45b6-ac8e-120633dd429e","Type":"ContainerStarted","Data":"8e02cdb15cc1a280ad489bdb7b0d875917c6d6bc254a259be81495d7027af9ab"} Aug 04 15:17:33.077536 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:33.077473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" event={"ID":"309833bf-fc35-445a-9c48-c31ccd4c1f2d","Type":"ContainerStarted","Data":"3eec587f6722a882d3cfdcc4ac35c01b62f5068e019b01f436b88d4df2beaa8d"} Aug 04 15:17:33.097870 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:33.097817 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86b9dff769-mctsj" podStartSLOduration=1.8547468299999998 podStartE2EDuration="5.097796653s" totalImagesPullingTime="3.243049823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:17:33.094796908 +0000 UTC m=+418.448884668" watchObservedRunningTime="2026-08-04 15:17:33.097796653 +0000 UTC m=+418.451884415" Aug 04 15:17:34.695339 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:34.695289 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:36.433306 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:36.433264 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Aug 04 15:17:36.433662 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:36.433345 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"} Aug 04 15:17:37.089910 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.089867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:37.100656 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.100620 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" event={"ID":"244b7eea-6747-4456-99d2-c82257b5bf6e","Type":"ContainerStarted","Data":"71eb213a0ff0a99b611cff3ebbc11543193e5ffaf602d40d559b0856d2220830"} Aug 04 15:17:37.100840 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.100769 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:37.102498 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.102464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" event={"ID":"1747d4d9-7474-45ae-8b9f-0889f79b22cd","Type":"ContainerStarted","Data":"1c4657dbf5226b3413e040085c45e2e4ee5f79897c11537393ff4efc00229b39"} Aug 04 15:17:37.102629 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.102516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:17:37.104220 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.104195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" event={"ID":"ac9c6835-dff6-45b6-ac8e-120633dd429e","Type":"ContainerStarted","Data":"e0fe165ce242cf834bcd58a286357c178289cbb87a82dbcf1dd9923e9d3de5d0"} Aug 04 15:17:37.104402 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.104388 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:37.106397 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.106374 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-tz428 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Aug 04 15:17:37.106493 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.106419 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" podUID="ac9c6835-dff6-45b6-ac8e-120633dd429e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Aug 04 15:17:37.124054 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.124001 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" podStartSLOduration=2.001375211 podStartE2EDuration="8.123987157s" totalImagesPullingTime="6.122611946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:17:37.122170433 +0000 UTC m=+422.476258192" watchObservedRunningTime="2026-08-04 15:17:37.123987157 +0000 UTC m=+422.478074954" Aug 04 15:17:37.149725 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.149669 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" podStartSLOduration=2.431319159 podStartE2EDuration="6.149652859s" totalImagesPullingTime="3.7183337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:17:37.147938905 +0000 UTC m=+422.502026665" watchObservedRunningTime="2026-08-04 15:17:37.149652859 +0000 UTC m=+422.503740617" Aug 04 15:17:37.172603 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.172508 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" podStartSLOduration=1.8663705240000001 podStartE2EDuration="7.172460418s" totalImagesPullingTime="5.306089894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:17:37.171988588 +0000 UTC m=+422.526076348" watchObservedRunningTime="2026-08-04 15:17:37.172460418 +0000 UTC m=+422.526548180" Aug 04 15:17:37.183942 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.183908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:37.775126 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:37.775071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:38.086058 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:38.085961 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:38.109535 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:38.109509 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:38.194003 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:38.193969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:40.611987 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.611932 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:40.686806 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.686773 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh"] Aug 04 15:17:40.690273 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.690255 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:40.692763 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.692737 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wvfbv\"" Aug 04 15:17:40.692763 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.692761 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Aug 04 15:17:40.702683 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.702662 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh"] Aug 04 15:17:40.720006 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.719978 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:40.775331 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.775290 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7jb\" (UniqueName: \"kubernetes.io/projected/d08d6054-e6eb-47ad-9ab7-42a5486668af-kube-api-access-lx7jb\") pod \"kserve-controller-manager-d6f4b65b8-vvhzh\" (UID: \"d08d6054-e6eb-47ad-9ab7-42a5486668af\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:40.775521 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.775409 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d08d6054-e6eb-47ad-9ab7-42a5486668af-cert\") pod \"kserve-controller-manager-d6f4b65b8-vvhzh\" (UID: \"d08d6054-e6eb-47ad-9ab7-42a5486668af\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:40.875935 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.875846 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d08d6054-e6eb-47ad-9ab7-42a5486668af-cert\") pod \"kserve-controller-manager-d6f4b65b8-vvhzh\" (UID: \"d08d6054-e6eb-47ad-9ab7-42a5486668af\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:40.875935 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.875894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7jb\" (UniqueName: \"kubernetes.io/projected/d08d6054-e6eb-47ad-9ab7-42a5486668af-kube-api-access-lx7jb\") pod \"kserve-controller-manager-d6f4b65b8-vvhzh\" (UID: \"d08d6054-e6eb-47ad-9ab7-42a5486668af\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:40.876129 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:17:40.876029 2559 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Aug 04 15:17:40.876129 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:17:40.876108 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08d6054-e6eb-47ad-9ab7-42a5486668af-cert podName:d08d6054-e6eb-47ad-9ab7-42a5486668af nodeName:}" failed. No retries permitted until 2026-08-04 15:17:41.376087235 +0000 UTC m=+426.730174978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d08d6054-e6eb-47ad-9ab7-42a5486668af-cert") pod "kserve-controller-manager-d6f4b65b8-vvhzh" (UID: "d08d6054-e6eb-47ad-9ab7-42a5486668af") : secret "kserve-webhook-server-cert" not found Aug 04 15:17:40.898166 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.898135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7jb\" (UniqueName: \"kubernetes.io/projected/d08d6054-e6eb-47ad-9ab7-42a5486668af-kube-api-access-lx7jb\") pod \"kserve-controller-manager-d6f4b65b8-vvhzh\" (UID: \"d08d6054-e6eb-47ad-9ab7-42a5486668af\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:40.906471 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:40.906422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:41.004319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:41.004285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:41.172223 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:41.172133 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:41.322272 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:41.322233 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:17:41.379755 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:41.379720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d08d6054-e6eb-47ad-9ab7-42a5486668af-cert\") pod \"kserve-controller-manager-d6f4b65b8-vvhzh\" (UID: \"d08d6054-e6eb-47ad-9ab7-42a5486668af\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:41.382194 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:41.382174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d08d6054-e6eb-47ad-9ab7-42a5486668af-cert\") pod \"kserve-controller-manager-d6f4b65b8-vvhzh\" (UID: \"d08d6054-e6eb-47ad-9ab7-42a5486668af\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:41.603639 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:41.603598 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:41.736371 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:17:41.736339 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08d6054_e6eb_47ad_9ab7_42a5486668af.slice/crio-79f3a9e3019cc20ee1f7c2a783522d4476626cca964bc44460766b0f11d7c428 WatchSource:0}: Error finding container 79f3a9e3019cc20ee1f7c2a783522d4476626cca964bc44460766b0f11d7c428: Status 404 returned error can't find the container with id 79f3a9e3019cc20ee1f7c2a783522d4476626cca964bc44460766b0f11d7c428 Aug 04 15:17:41.738628 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:41.738384 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh"] Aug 04 15:17:42.127211 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:42.127174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" event={"ID":"d08d6054-e6eb-47ad-9ab7-42a5486668af","Type":"ContainerStarted","Data":"79f3a9e3019cc20ee1f7c2a783522d4476626cca964bc44460766b0f11d7c428"} Aug 04 15:17:43.665187 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:43.665148 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428"] Aug 04 15:17:43.665623 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:43.665325 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" podUID="ac9c6835-dff6-45b6-ac8e-120633dd429e" containerName="discovery" containerID="cri-o://e0fe165ce242cf834bcd58a286357c178289cbb87a82dbcf1dd9923e9d3de5d0" gracePeriod=30 Aug 04 15:17:44.109819 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.109780 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-tz428 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/ready\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body= Aug 04 15:17:44.109937 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.109849 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" podUID="ac9c6835-dff6-45b6-ac8e-120633dd429e" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.29:8080/ready\": dial tcp 10.132.0.29:8080: connect: connection refused" Aug 04 15:17:44.137107 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.137076 2559 generic.go:356] "Generic (PLEG): container finished" podID="ac9c6835-dff6-45b6-ac8e-120633dd429e" containerID="e0fe165ce242cf834bcd58a286357c178289cbb87a82dbcf1dd9923e9d3de5d0" exitCode=0 Aug 04 15:17:44.137267 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.137143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" event={"ID":"ac9c6835-dff6-45b6-ac8e-120633dd429e","Type":"ContainerDied","Data":"e0fe165ce242cf834bcd58a286357c178289cbb87a82dbcf1dd9923e9d3de5d0"} Aug 04 15:17:44.228865 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.228839 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:44.299321 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299287 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-kubeconfig\") pod \"ac9c6835-dff6-45b6-ac8e-120633dd429e\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " Aug 04 15:17:44.299461 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299330 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-dns-cert\") pod \"ac9c6835-dff6-45b6-ac8e-120633dd429e\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " Aug 04 15:17:44.299461 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299360 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d87qd\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-kube-api-access-d87qd\") pod \"ac9c6835-dff6-45b6-ac8e-120633dd429e\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " Aug 04 15:17:44.299563 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299471 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-ca-configmap\") pod \"ac9c6835-dff6-45b6-ac8e-120633dd429e\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " Aug 04 15:17:44.299563 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299514 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9c6835-dff6-45b6-ac8e-120633dd429e-local-certs\") pod \"ac9c6835-dff6-45b6-ac8e-120633dd429e\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " Aug 04 15:17:44.299563 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299547 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-token\") pod \"ac9c6835-dff6-45b6-ac8e-120633dd429e\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " Aug 04 15:17:44.299747 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299638 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-cacerts\") pod \"ac9c6835-dff6-45b6-ac8e-120633dd429e\" (UID: \"ac9c6835-dff6-45b6-ac8e-120633dd429e\") " Aug 04 15:17:44.299992 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299823 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ac9c6835-dff6-45b6-ac8e-120633dd429e" (UID: "ac9c6835-dff6-45b6-ac8e-120633dd429e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:17:44.299992 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.299954 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-ca-configmap\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:17:44.302143 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.302111 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac9c6835-dff6-45b6-ac8e-120633dd429e" (UID: "ac9c6835-dff6-45b6-ac8e-120633dd429e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:17:44.302308 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.302284 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-cacerts" (OuterVolumeSpecName: "cacerts") pod "ac9c6835-dff6-45b6-ac8e-120633dd429e" (UID: "ac9c6835-dff6-45b6-ac8e-120633dd429e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:17:44.302459 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.302323 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ac9c6835-dff6-45b6-ac8e-120633dd429e" (UID: "ac9c6835-dff6-45b6-ac8e-120633dd429e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:17:44.302459 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.302403 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-kube-api-access-d87qd" (OuterVolumeSpecName: "kube-api-access-d87qd") pod "ac9c6835-dff6-45b6-ac8e-120633dd429e" (UID: "ac9c6835-dff6-45b6-ac8e-120633dd429e"). InnerVolumeSpecName "kube-api-access-d87qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:17:44.302694 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.302672 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9c6835-dff6-45b6-ac8e-120633dd429e-local-certs" (OuterVolumeSpecName: "local-certs") pod "ac9c6835-dff6-45b6-ac8e-120633dd429e" (UID: "ac9c6835-dff6-45b6-ac8e-120633dd429e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 15:17:44.302954 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.302930 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ac9c6835-dff6-45b6-ac8e-120633dd429e" (UID: "ac9c6835-dff6-45b6-ac8e-120633dd429e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:17:44.400700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.400600 2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9c6835-dff6-45b6-ac8e-120633dd429e-local-certs\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:17:44.400700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.400646 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-token\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:17:44.400700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.400656 2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-cacerts\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:17:44.400700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.400664 2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-kubeconfig\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:17:44.400700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.400672 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac9c6835-dff6-45b6-ac8e-120633dd429e-istio-csr-dns-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:17:44.400700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:44.400682 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d87qd\" (UniqueName: \"kubernetes.io/projected/ac9c6835-dff6-45b6-ac8e-120633dd429e-kube-api-access-d87qd\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:17:45.142555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.142524 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" Aug 04 15:17:45.143052 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.142520 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428" event={"ID":"ac9c6835-dff6-45b6-ac8e-120633dd429e","Type":"ContainerDied","Data":"8e02cdb15cc1a280ad489bdb7b0d875917c6d6bc254a259be81495d7027af9ab"} Aug 04 15:17:45.143052 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.142633 2559 scope.go:117] "RemoveContainer" containerID="e0fe165ce242cf834bcd58a286357c178289cbb87a82dbcf1dd9923e9d3de5d0" Aug 04 15:17:45.144138 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.144115 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" event={"ID":"d08d6054-e6eb-47ad-9ab7-42a5486668af","Type":"ContainerStarted","Data":"8334cde608d2d47bd47a255cade57088cd5086b5b4ff3b94a324fc4bf5a2e7c6"} Aug 04 15:17:45.144285 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.144266 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:17:45.165528 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.165483 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" podStartSLOduration=2.759874945 podStartE2EDuration="5.16547066s" totalImagesPullingTime="2.405595715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:17:45.163657256 +0000 UTC m=+430.517745014" watchObservedRunningTime="2026-08-04 15:17:45.16547066 +0000 UTC m=+430.519558420" Aug 04 15:17:45.179707 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.179684 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428"] Aug 04 15:17:45.183778 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.183756 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-tz428"] Aug 04 15:17:45.287391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:45.287357 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac9c6835-dff6-45b6-ac8e-120633dd429e" path="/var/lib/kubelet/pods/ac9c6835-dff6-45b6-ac8e-120633dd429e/volumes" Aug 04 15:17:48.111735 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:48.111699 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-jtsss" Aug 04 15:17:48.114810 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:17:48.114790 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-bd7pk" Aug 04 15:18:16.154331 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:16.154250 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vvhzh" Aug 04 15:18:30.217207 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:30.217170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:18:31.033502 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:31.033455 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:18:43.066001 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:43.065964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:18:54.724260 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:54.724216 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:18:56.786616 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.786567 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54"] Aug 04 15:18:56.787070 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.787050 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9c6835-dff6-45b6-ac8e-120633dd429e" containerName="discovery" Aug 04 15:18:56.787144 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.787073 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9c6835-dff6-45b6-ac8e-120633dd429e" containerName="discovery" Aug 04 15:18:56.787196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.787188 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9c6835-dff6-45b6-ac8e-120633dd429e" containerName="discovery" Aug 04 15:18:56.789980 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.789959 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:18:56.794359 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.794336 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Aug 04 15:18:56.794441 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.794382 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Aug 04 15:18:56.794908 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.794893 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bpvhx\"" Aug 04 15:18:56.810196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.810173 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54"] Aug 04 15:18:56.921289 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:56.921226 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56d8\" (UniqueName: \"kubernetes.io/projected/938b5cbb-e3fd-4b33-9a8d-56aea5c4d517-kube-api-access-w56d8\") pod \"limitador-operator-controller-manager-6c549b6d85-28l54\" (UID: \"938b5cbb-e3fd-4b33-9a8d-56aea5c4d517\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:18:57.022616 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:57.022558 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56d8\" (UniqueName: \"kubernetes.io/projected/938b5cbb-e3fd-4b33-9a8d-56aea5c4d517-kube-api-access-w56d8\") pod \"limitador-operator-controller-manager-6c549b6d85-28l54\" (UID: \"938b5cbb-e3fd-4b33-9a8d-56aea5c4d517\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:18:57.037246 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:57.037177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56d8\" (UniqueName: \"kubernetes.io/projected/938b5cbb-e3fd-4b33-9a8d-56aea5c4d517-kube-api-access-w56d8\") pod \"limitador-operator-controller-manager-6c549b6d85-28l54\" (UID: \"938b5cbb-e3fd-4b33-9a8d-56aea5c4d517\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:18:57.100264 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:57.100221 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:18:57.232520 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:57.232443 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54"] Aug 04 15:18:57.235479 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:18:57.235450 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938b5cbb_e3fd_4b33_9a8d_56aea5c4d517.slice/crio-094da9dcc5d5ff173bcda219f00a0c3f3561cf73c2c534becf60b7453bd5ee25 WatchSource:0}: Error finding container 094da9dcc5d5ff173bcda219f00a0c3f3561cf73c2c534becf60b7453bd5ee25: Status 404 returned error can't find the container with id 094da9dcc5d5ff173bcda219f00a0c3f3561cf73c2c534becf60b7453bd5ee25 Aug 04 15:18:57.457471 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:57.457438 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" event={"ID":"938b5cbb-e3fd-4b33-9a8d-56aea5c4d517","Type":"ContainerStarted","Data":"094da9dcc5d5ff173bcda219f00a0c3f3561cf73c2c534becf60b7453bd5ee25"} Aug 04 15:18:57.552700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:57.552666 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:18:58.796749 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:58.796398 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jq74f"] Aug 04 15:18:58.799231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:58.799203 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" Aug 04 15:18:58.802184 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:58.802162 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bpblm\"" Aug 04 15:18:58.802680 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:58.802660 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Aug 04 15:18:58.809481 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:58.809461 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jq74f"] Aug 04 15:18:58.940925 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:58.940883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86vw\" (UniqueName: \"kubernetes.io/projected/0ea2134c-e8d7-4dd1-99b4-c75708169752-kube-api-access-w86vw\") pod \"dns-operator-controller-manager-5985564776-jq74f\" (UID: \"0ea2134c-e8d7-4dd1-99b4-c75708169752\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" Aug 04 15:18:59.042169 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:59.042134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86vw\" (UniqueName: \"kubernetes.io/projected/0ea2134c-e8d7-4dd1-99b4-c75708169752-kube-api-access-w86vw\") pod \"dns-operator-controller-manager-5985564776-jq74f\" (UID: \"0ea2134c-e8d7-4dd1-99b4-c75708169752\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" Aug 04 15:18:59.073265 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:59.073191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86vw\" (UniqueName: \"kubernetes.io/projected/0ea2134c-e8d7-4dd1-99b4-c75708169752-kube-api-access-w86vw\") pod \"dns-operator-controller-manager-5985564776-jq74f\" (UID: \"0ea2134c-e8d7-4dd1-99b4-c75708169752\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" Aug 04 15:18:59.113295 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:59.113084 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" Aug 04 15:18:59.993317 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:18:59.993243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:00.031089 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:00.031067 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jq74f"] Aug 04 15:19:00.033407 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:00.033379 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea2134c_e8d7_4dd1_99b4_c75708169752.slice/crio-c7ca8f82b519804d52dc44d2879f87e55a98b6adadf8b17b378d4fae6f21ddd5 WatchSource:0}: Error finding container c7ca8f82b519804d52dc44d2879f87e55a98b6adadf8b17b378d4fae6f21ddd5: Status 404 returned error can't find the container with id c7ca8f82b519804d52dc44d2879f87e55a98b6adadf8b17b378d4fae6f21ddd5 Aug 04 15:19:00.090753 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:00.090706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:00.200403 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:00.200364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:00.481249 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:00.481128 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" event={"ID":"938b5cbb-e3fd-4b33-9a8d-56aea5c4d517","Type":"ContainerStarted","Data":"833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1"} Aug 04 15:19:00.481249 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:00.481197 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:00.482445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:00.482421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" event={"ID":"0ea2134c-e8d7-4dd1-99b4-c75708169752","Type":"ContainerStarted","Data":"c7ca8f82b519804d52dc44d2879f87e55a98b6adadf8b17b378d4fae6f21ddd5"} Aug 04 15:19:00.516616 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:00.516554 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" podStartSLOduration=1.817318413 podStartE2EDuration="4.516540753s" totalImagesPullingTime="2.69922234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:00.512229217 +0000 UTC m=+505.866316976" watchObservedRunningTime="2026-08-04 15:19:00.516540753 +0000 UTC m=+505.870628512" Aug 04 15:19:01.194393 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:01.194349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:02.199484 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:02.199434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:03.371864 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.371832 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn"] Aug 04 15:19:03.374278 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.374257 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.376679 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.376653 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-85tjv\"" Aug 04 15:19:03.390571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.390544 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn"] Aug 04 15:19:03.482738 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.482684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dfda0c-00cb-45a6-9803-d689f417fa5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.482738 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.482731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/23dfda0c-00cb-45a6-9803-d689f417fa5a-kube-api-access-j8jr9\") pod \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.515841 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.515805 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:03.583671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.583635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dfda0c-00cb-45a6-9803-d689f417fa5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.583671 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.583674 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/23dfda0c-00cb-45a6-9803-d689f417fa5a-kube-api-access-j8jr9\") pod \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.584083 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.584065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dfda0c-00cb-45a6-9803-d689f417fa5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.592214 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.592180 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/23dfda0c-00cb-45a6-9803-d689f417fa5a-kube-api-access-j8jr9\") pod \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.684433 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.684356 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:03.758194 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.758163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:03.820895 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.820858 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn"] Aug 04 15:19:03.823795 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:03.823769 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dfda0c_00cb_45a6_9803_d689f417fa5a.slice/crio-eff5ccb0696278e3488abfc501a8441bf7f5a319c19c273b84f9bc6ef1fe1c79 WatchSource:0}: Error finding container eff5ccb0696278e3488abfc501a8441bf7f5a319c19c273b84f9bc6ef1fe1c79: Status 404 returned error can't find the container with id eff5ccb0696278e3488abfc501a8441bf7f5a319c19c273b84f9bc6ef1fe1c79 Aug 04 15:19:03.923075 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:03.923029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:04.024120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:04.024085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:04.501401 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:04.501366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" event={"ID":"23dfda0c-00cb-45a6-9803-d689f417fa5a","Type":"ContainerStarted","Data":"eff5ccb0696278e3488abfc501a8441bf7f5a319c19c273b84f9bc6ef1fe1c79"} Aug 04 15:19:06.054201 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.054165 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-q4vw5"] Aug 04 15:19:06.056915 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.056894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" Aug 04 15:19:06.060140 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.060103 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-92z8j\"" Aug 04 15:19:06.072372 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.072349 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-q4vw5"] Aug 04 15:19:06.207736 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.207702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r4hd\" (UniqueName: \"kubernetes.io/projected/a7a0fe87-bd13-424d-b988-9d10c5801af5-kube-api-access-9r4hd\") pod \"authorino-operator-68d88948d8-q4vw5\" (UID: \"a7a0fe87-bd13-424d-b988-9d10c5801af5\") " pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" Aug 04 15:19:06.309188 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.309109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r4hd\" (UniqueName: \"kubernetes.io/projected/a7a0fe87-bd13-424d-b988-9d10c5801af5-kube-api-access-9r4hd\") pod \"authorino-operator-68d88948d8-q4vw5\" (UID: \"a7a0fe87-bd13-424d-b988-9d10c5801af5\") " pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" Aug 04 15:19:06.324634 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.324601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r4hd\" (UniqueName: \"kubernetes.io/projected/a7a0fe87-bd13-424d-b988-9d10c5801af5-kube-api-access-9r4hd\") pod \"authorino-operator-68d88948d8-q4vw5\" (UID: \"a7a0fe87-bd13-424d-b988-9d10c5801af5\") " pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" Aug 04 15:19:06.369193 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.369164 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" Aug 04 15:19:06.511709 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:06.511686 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-q4vw5"] Aug 04 15:19:06.514416 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:06.514392 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a0fe87_bd13_424d_b988_9d10c5801af5.slice/crio-8196fef62142478b6dde29514392c36ec8ab70a1720b14b214bc4923ad63b3c6 WatchSource:0}: Error finding container 8196fef62142478b6dde29514392c36ec8ab70a1720b14b214bc4923ad63b3c6: Status 404 returned error can't find the container with id 8196fef62142478b6dde29514392c36ec8ab70a1720b14b214bc4923ad63b3c6 Aug 04 15:19:07.329890 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:07.329853 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:07.434904 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:07.434870 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:07.519899 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:07.519862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" event={"ID":"a7a0fe87-bd13-424d-b988-9d10c5801af5","Type":"ContainerStarted","Data":"8196fef62142478b6dde29514392c36ec8ab70a1720b14b214bc4923ad63b3c6"} Aug 04 15:19:09.531133 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:09.531046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" event={"ID":"23dfda0c-00cb-45a6-9803-d689f417fa5a","Type":"ContainerStarted","Data":"4770f90940233fd78786010994ddfcd5637e820bcde75c6b6c572cea8685301e"} Aug 04 15:19:09.531569 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:09.531213 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:09.532834 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:09.532810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" event={"ID":"a7a0fe87-bd13-424d-b988-9d10c5801af5","Type":"ContainerStarted","Data":"14ed626324109d5fd68ccfe223cbb373ececd803b9f9a92fe57997b96c28c7b0"} Aug 04 15:19:09.532951 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:09.532909 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" Aug 04 15:19:09.553715 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:09.553661 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" podStartSLOduration=1.107411589 podStartE2EDuration="6.553644596s" totalImagesPullingTime="5.446233007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:09.552129588 +0000 UTC m=+514.906217349" watchObservedRunningTime="2026-08-04 15:19:09.553644596 +0000 UTC m=+514.907732354" Aug 04 15:19:09.571985 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:09.571927 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" podStartSLOduration=0.828472786 podStartE2EDuration="3.57190955s" totalImagesPullingTime="2.743436764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:19:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:09.569874542 +0000 UTC m=+514.923962294" watchObservedRunningTime="2026-08-04 15:19:09.57190955 +0000 UTC m=+514.925997308" Aug 04 15:19:10.337217 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:10.337176 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:10.437167 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:10.437131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:10.544299 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:10.544192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" event={"ID":"0ea2134c-e8d7-4dd1-99b4-c75708169752","Type":"ContainerStarted","Data":"3c809a3837a14917a1358e979f9e5c2cc08e8d827b8c0d3c470bc09dec23da50"} Aug 04 15:19:10.563440 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:10.563385 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" podStartSLOduration=2.358544881 podStartE2EDuration="12.563368847s" totalImagesPullingTime="10.204823966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:18:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:10.562320205 +0000 UTC m=+515.916407965" watchObservedRunningTime="2026-08-04 15:19:10.563368847 +0000 UTC m=+515.917456605" Aug 04 15:19:11.489311 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:11.489278 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:11.549512 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:11.549479 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" Aug 04 15:19:12.110018 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:12.109987 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:12.442570 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:12.442542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:15.307098 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:15.307057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:15.404913 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:15.404878 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:15.507018 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:15.506981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:20.546257 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:20.546227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:20.546690 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:20.546294 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-q4vw5" Aug 04 15:19:22.479738 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.479689 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn"] Aug 04 15:19:22.480145 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.479888 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" containerName="manager" containerID="cri-o://4770f90940233fd78786010994ddfcd5637e820bcde75c6b6c572cea8685301e" gracePeriod=2 Aug 04 15:19:22.490668 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.490639 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn"] Aug 04 15:19:22.505668 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.505617 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"23dfda0c-00cb-45a6-9803-d689f417fa5a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-dvcnn\": pods \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.507382 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.507351 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54"] Aug 04 15:19:22.507763 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.507567 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" containerName="manager" containerID="cri-o://833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1" gracePeriod=2 Aug 04 15:19:22.509687 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.509647 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-28l54\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:22.509840 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.509802 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx"] Aug 04 15:19:22.510438 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.510274 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" containerName="manager" Aug 04 15:19:22.510438 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.510293 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" containerName="manager" Aug 04 15:19:22.510438 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.510390 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" containerName="manager" Aug 04 15:19:22.511829 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.511798 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.512915 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.512896 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54"] Aug 04 15:19:22.513018 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.512998 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.515110 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.515070 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.536591 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.536554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx"] Aug 04 15:19:22.547073 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.547046 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl"] Aug 04 15:19:22.547402 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.547387 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" containerName="manager" Aug 04 15:19:22.547402 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.547402 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" containerName="manager" Aug 04 15:19:22.547509 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.547456 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" containerName="manager" Aug 04 15:19:22.551764 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.551731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69kr7\" (UniqueName: \"kubernetes.io/projected/313f053b-53a1-4927-bc3d-de18b98fe3f3-kube-api-access-69kr7\") pod \"kuadrant-operator-controller-manager-58fc56757c-bvvnx\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.551868 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.551781 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f053b-53a1-4927-bc3d-de18b98fe3f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bvvnx\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.553666 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.553634 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-28l54\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:22.554968 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.554939 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" Aug 04 15:19:22.556411 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.556392 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jq74f" Aug 04 15:19:22.557355 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.557327 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.563128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.563104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl"] Aug 04 15:19:22.579468 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.579434 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-28l54\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:22.581888 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.581858 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.583637 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.583618 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-28l54\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:22.601496 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.601472 2559 generic.go:356] "Generic (PLEG): container finished" podID="23dfda0c-00cb-45a6-9803-d689f417fa5a" containerID="4770f90940233fd78786010994ddfcd5637e820bcde75c6b6c572cea8685301e" exitCode=0 Aug 04 15:19:22.652970 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.652925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69kr7\" (UniqueName: \"kubernetes.io/projected/313f053b-53a1-4927-bc3d-de18b98fe3f3-kube-api-access-69kr7\") pod \"kuadrant-operator-controller-manager-58fc56757c-bvvnx\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.653138 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.652974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9mk\" (UniqueName: \"kubernetes.io/projected/1df49bf1-def8-4a17-8d71-8a567366f1b0-kube-api-access-gn9mk\") pod \"limitador-operator-controller-manager-6c549b6d85-r92wl\" (UID: \"1df49bf1-def8-4a17-8d71-8a567366f1b0\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" Aug 04 15:19:22.653138 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.653007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f053b-53a1-4927-bc3d-de18b98fe3f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bvvnx\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.653509 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.653484 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f053b-53a1-4927-bc3d-de18b98fe3f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bvvnx\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.662920 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.662894 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69kr7\" (UniqueName: \"kubernetes.io/projected/313f053b-53a1-4927-bc3d-de18b98fe3f3-kube-api-access-69kr7\") pod \"kuadrant-operator-controller-manager-58fc56757c-bvvnx\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.736463 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.736435 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n"] Aug 04 15:19:22.739446 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.739424 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:22.749921 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.749900 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.753457 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.753435 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/23dfda0c-00cb-45a6-9803-d689f417fa5a-kube-api-access-j8jr9\") pod \"23dfda0c-00cb-45a6-9803-d689f417fa5a\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " Aug 04 15:19:22.753557 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.753513 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dfda0c-00cb-45a6-9803-d689f417fa5a-extensions-socket-volume\") pod \"23dfda0c-00cb-45a6-9803-d689f417fa5a\" (UID: \"23dfda0c-00cb-45a6-9803-d689f417fa5a\") " Aug 04 15:19:22.753714 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.753695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e73eaf0-0e1f-4618-9a2e-bb199588548d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-s6s2n\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:22.753815 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.753789 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9mk\" (UniqueName: \"kubernetes.io/projected/1df49bf1-def8-4a17-8d71-8a567366f1b0-kube-api-access-gn9mk\") pod \"limitador-operator-controller-manager-6c549b6d85-r92wl\" (UID: \"1df49bf1-def8-4a17-8d71-8a567366f1b0\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" Aug 04 15:19:22.753959 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.753892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8qg\" (UniqueName: \"kubernetes.io/projected/9e73eaf0-0e1f-4618-9a2e-bb199588548d-kube-api-access-4w8qg\") pod \"kuadrant-operator-controller-manager-68cf589589-s6s2n\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:22.754098 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.754074 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23dfda0c-00cb-45a6-9803-d689f417fa5a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "23dfda0c-00cb-45a6-9803-d689f417fa5a" (UID: "23dfda0c-00cb-45a6-9803-d689f417fa5a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 15:19:22.754970 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.754890 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n"] Aug 04 15:19:22.756017 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.755996 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23dfda0c-00cb-45a6-9803-d689f417fa5a-kube-api-access-j8jr9" (OuterVolumeSpecName: "kube-api-access-j8jr9") pod "23dfda0c-00cb-45a6-9803-d689f417fa5a" (UID: "23dfda0c-00cb-45a6-9803-d689f417fa5a"). InnerVolumeSpecName "kube-api-access-j8jr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:19:22.762116 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.762093 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:22.770520 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.770501 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9mk\" (UniqueName: \"kubernetes.io/projected/1df49bf1-def8-4a17-8d71-8a567366f1b0-kube-api-access-gn9mk\") pod \"limitador-operator-controller-manager-6c549b6d85-r92wl\" (UID: \"1df49bf1-def8-4a17-8d71-8a567366f1b0\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" Aug 04 15:19:22.785279 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.785255 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-28l54\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:22.787462 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.787439 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.789698 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.789675 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-dvcnn\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:22.791519 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:22.791498 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-28l54\" is forbidden: User \"system:node:ip-10-0-138-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-35.ec2.internal' and this object" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:22.854980 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.854947 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w56d8\" (UniqueName: \"kubernetes.io/projected/938b5cbb-e3fd-4b33-9a8d-56aea5c4d517-kube-api-access-w56d8\") pod \"938b5cbb-e3fd-4b33-9a8d-56aea5c4d517\" (UID: \"938b5cbb-e3fd-4b33-9a8d-56aea5c4d517\") " Aug 04 15:19:22.855122 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.855102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8qg\" (UniqueName: \"kubernetes.io/projected/9e73eaf0-0e1f-4618-9a2e-bb199588548d-kube-api-access-4w8qg\") pod \"kuadrant-operator-controller-manager-68cf589589-s6s2n\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:22.855189 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.855141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e73eaf0-0e1f-4618-9a2e-bb199588548d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-s6s2n\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:22.855245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.855222 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8jr9\" (UniqueName: \"kubernetes.io/projected/23dfda0c-00cb-45a6-9803-d689f417fa5a-kube-api-access-j8jr9\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:19:22.855245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.855241 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23dfda0c-00cb-45a6-9803-d689f417fa5a-extensions-socket-volume\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:19:22.855534 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.855510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e73eaf0-0e1f-4618-9a2e-bb199588548d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-s6s2n\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:22.856980 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.856959 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938b5cbb-e3fd-4b33-9a8d-56aea5c4d517-kube-api-access-w56d8" (OuterVolumeSpecName: "kube-api-access-w56d8") pod "938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" (UID: "938b5cbb-e3fd-4b33-9a8d-56aea5c4d517"). InnerVolumeSpecName "kube-api-access-w56d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:19:22.868451 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.868430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8qg\" (UniqueName: \"kubernetes.io/projected/9e73eaf0-0e1f-4618-9a2e-bb199588548d-kube-api-access-4w8qg\") pod \"kuadrant-operator-controller-manager-68cf589589-s6s2n\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:22.918011 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.917982 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:22.926057 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.926034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" Aug 04 15:19:22.955906 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:22.955874 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56d8\" (UniqueName: \"kubernetes.io/projected/938b5cbb-e3fd-4b33-9a8d-56aea5c4d517-kube-api-access-w56d8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:19:23.059300 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.059270 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:23.071877 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.071826 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx"] Aug 04 15:19:23.074167 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:23.074139 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313f053b_53a1_4927_bc3d_de18b98fe3f3.slice/crio-6b249361392c3a9fff036889c539c443d1e41fa853add483bf444455f8fb6575 WatchSource:0}: Error finding container 6b249361392c3a9fff036889c539c443d1e41fa853add483bf444455f8fb6575: Status 404 returned error can't find the container with id 6b249361392c3a9fff036889c539c443d1e41fa853add483bf444455f8fb6575 Aug 04 15:19:23.094802 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.094781 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl"] Aug 04 15:19:23.097787 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:23.097759 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df49bf1_def8_4a17_8d71_8a567366f1b0.slice/crio-5daf646dc379fb66e0af3197c5d9ecf5cf636df06735a1e17859968a3fb6c856 WatchSource:0}: Error finding container 5daf646dc379fb66e0af3197c5d9ecf5cf636df06735a1e17859968a3fb6c856: Status 404 returned error can't find the container with id 5daf646dc379fb66e0af3197c5d9ecf5cf636df06735a1e17859968a3fb6c856 Aug 04 15:19:23.210821 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.210796 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n"] Aug 04 15:19:23.212964 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:23.212936 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e73eaf0_0e1f_4618_9a2e_bb199588548d.slice/crio-498e68d00e5029a80967a4d5300e4c04b7e87e833b5dbf3461c8d850f97f167f WatchSource:0}: Error finding container 498e68d00e5029a80967a4d5300e4c04b7e87e833b5dbf3461c8d850f97f167f: Status 404 returned error can't find the container with id 498e68d00e5029a80967a4d5300e4c04b7e87e833b5dbf3461c8d850f97f167f Aug 04 15:19:23.289563 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.289535 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23dfda0c-00cb-45a6-9803-d689f417fa5a" path="/var/lib/kubelet/pods/23dfda0c-00cb-45a6-9803-d689f417fa5a/volumes" Aug 04 15:19:23.289944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.289927 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" path="/var/lib/kubelet/pods/938b5cbb-e3fd-4b33-9a8d-56aea5c4d517/volumes" Aug 04 15:19:23.607364 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.607283 2559 generic.go:356] "Generic (PLEG): container finished" podID="938b5cbb-e3fd-4b33-9a8d-56aea5c4d517" containerID="833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1" exitCode=0 Aug 04 15:19:23.607364 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.607337 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-28l54" Aug 04 15:19:23.607889 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.607391 2559 scope.go:117] "RemoveContainer" containerID="833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1" Aug 04 15:19:23.609195 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.609163 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" event={"ID":"9e73eaf0-0e1f-4618-9a2e-bb199588548d","Type":"ContainerStarted","Data":"a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180"} Aug 04 15:19:23.609300 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.609259 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" event={"ID":"9e73eaf0-0e1f-4618-9a2e-bb199588548d","Type":"ContainerStarted","Data":"498e68d00e5029a80967a4d5300e4c04b7e87e833b5dbf3461c8d850f97f167f"} Aug 04 15:19:23.609468 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.609449 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:23.610854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.610838 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-dvcnn" Aug 04 15:19:23.612511 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.612485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" event={"ID":"1df49bf1-def8-4a17-8d71-8a567366f1b0","Type":"ContainerStarted","Data":"b2b8be7141448c9a9ffe6e36a4b69b37ccc0dfaef8d87a48c400596379d88c7e"} Aug 04 15:19:23.612612 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.612531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" event={"ID":"1df49bf1-def8-4a17-8d71-8a567366f1b0","Type":"ContainerStarted","Data":"5daf646dc379fb66e0af3197c5d9ecf5cf636df06735a1e17859968a3fb6c856"} Aug 04 15:19:23.612612 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.612560 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" Aug 04 15:19:23.614455 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.614434 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" event={"ID":"313f053b-53a1-4927-bc3d-de18b98fe3f3","Type":"ContainerStarted","Data":"e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd"} Aug 04 15:19:23.614555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.614474 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" event={"ID":"313f053b-53a1-4927-bc3d-de18b98fe3f3","Type":"ContainerStarted","Data":"6b249361392c3a9fff036889c539c443d1e41fa853add483bf444455f8fb6575"} Aug 04 15:19:23.614555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.614493 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:23.617228 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.617213 2559 scope.go:117] "RemoveContainer" containerID="833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1" Aug 04 15:19:23.617463 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:23.617445 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1\": container with ID starting with 833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1 not found: ID does not exist" containerID="833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1" Aug 04 15:19:23.617518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.617470 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1"} err="failed to get container status \"833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1\": rpc error: code = NotFound desc = could not find container \"833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1\": container with ID starting with 833baee81173f9bb0c0d0d10a45fb62abbeebdcbbfe1d7f2575a9032085af6a1 not found: ID does not exist" Aug 04 15:19:23.617518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.617486 2559 scope.go:117] "RemoveContainer" containerID="4770f90940233fd78786010994ddfcd5637e820bcde75c6b6c572cea8685301e" Aug 04 15:19:23.632337 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.632299 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" podStartSLOduration=1.63228769 podStartE2EDuration="1.63228769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:23.630349789 +0000 UTC m=+528.984437548" watchObservedRunningTime="2026-08-04 15:19:23.63228769 +0000 UTC m=+528.986375449" Aug 04 15:19:23.654232 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.654188 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" podStartSLOduration=1.654176418 podStartE2EDuration="1.654176418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:23.652312179 +0000 UTC m=+529.006399950" watchObservedRunningTime="2026-08-04 15:19:23.654176418 +0000 UTC m=+529.008264177" Aug 04 15:19:23.673980 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:23.673937 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" podStartSLOduration=1.6739241470000001 podStartE2EDuration="1.673924147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:23.672929374 +0000 UTC m=+529.027017134" watchObservedRunningTime="2026-08-04 15:19:23.673924147 +0000 UTC m=+529.028011905" Aug 04 15:19:24.331268 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:24.331223 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:24.575853 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:24.575802 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:26.797019 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:26.796967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:26.945305 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:26.945274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:29.800021 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:29.799939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:29.900730 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:29.900690 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:34.519924 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:34.519872 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:34.620546 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:34.620502 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:34.624011 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:34.623987 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:34.624152 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:34.624088 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-r92wl" Aug 04 15:19:34.624152 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:34.624138 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:19:34.735909 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:34.735161 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx"] Aug 04 15:19:34.736822 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:34.736104 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" podUID="313f053b-53a1-4927-bc3d-de18b98fe3f3" containerName="manager" containerID="cri-o://e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd" gracePeriod=10 Aug 04 15:19:35.021131 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.021099 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:35.060803 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.060735 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f053b-53a1-4927-bc3d-de18b98fe3f3-extensions-socket-volume\") pod \"313f053b-53a1-4927-bc3d-de18b98fe3f3\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " Aug 04 15:19:35.060803 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.060782 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69kr7\" (UniqueName: \"kubernetes.io/projected/313f053b-53a1-4927-bc3d-de18b98fe3f3-kube-api-access-69kr7\") pod \"313f053b-53a1-4927-bc3d-de18b98fe3f3\" (UID: \"313f053b-53a1-4927-bc3d-de18b98fe3f3\") " Aug 04 15:19:35.061183 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.061149 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313f053b-53a1-4927-bc3d-de18b98fe3f3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "313f053b-53a1-4927-bc3d-de18b98fe3f3" (UID: "313f053b-53a1-4927-bc3d-de18b98fe3f3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 15:19:35.062790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.062760 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313f053b-53a1-4927-bc3d-de18b98fe3f3-kube-api-access-69kr7" (OuterVolumeSpecName: "kube-api-access-69kr7") pod "313f053b-53a1-4927-bc3d-de18b98fe3f3" (UID: "313f053b-53a1-4927-bc3d-de18b98fe3f3"). InnerVolumeSpecName "kube-api-access-69kr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:19:35.161868 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.161838 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/313f053b-53a1-4927-bc3d-de18b98fe3f3-extensions-socket-volume\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:19:35.161868 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.161866 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69kr7\" (UniqueName: \"kubernetes.io/projected/313f053b-53a1-4927-bc3d-de18b98fe3f3-kube-api-access-69kr7\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:19:35.356633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.356536 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:35.456249 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.456213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:35.677537 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.677456 2559 generic.go:356] "Generic (PLEG): container finished" podID="313f053b-53a1-4927-bc3d-de18b98fe3f3" containerID="e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd" exitCode=0 Aug 04 15:19:35.677537 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.677501 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" Aug 04 15:19:35.677988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.677541 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" event={"ID":"313f053b-53a1-4927-bc3d-de18b98fe3f3","Type":"ContainerDied","Data":"e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd"} Aug 04 15:19:35.677988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.677616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx" event={"ID":"313f053b-53a1-4927-bc3d-de18b98fe3f3","Type":"ContainerDied","Data":"6b249361392c3a9fff036889c539c443d1e41fa853add483bf444455f8fb6575"} Aug 04 15:19:35.677988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.677643 2559 scope.go:117] "RemoveContainer" containerID="e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd" Aug 04 15:19:35.685731 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.685713 2559 scope.go:117] "RemoveContainer" containerID="e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd" Aug 04 15:19:35.685997 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:35.685979 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd\": container with ID starting with e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd not found: ID does not exist" containerID="e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd" Aug 04 15:19:35.686043 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.686006 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd"} err="failed to get container status \"e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd\": rpc error: code = NotFound desc = could not find container \"e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd\": container with ID starting with e9844c1491cbdb4d0d0932004fe9b0fb646a48f250d1356aa90918df99ec56dd not found: ID does not exist" Aug 04 15:19:35.702520 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.702496 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx"] Aug 04 15:19:35.708722 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:35.708699 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bvvnx"] Aug 04 15:19:36.349921 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:36.349887 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:36.450780 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:36.450735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:37.288330 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:37.288294 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="313f053b-53a1-4927-bc3d-de18b98fe3f3" path="/var/lib/kubelet/pods/313f053b-53a1-4927-bc3d-de18b98fe3f3/volumes" Aug 04 15:19:43.348918 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:43.348883 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:43.483222 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:43.483191 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:51.112944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:51.112907 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:51.829988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:51.829958 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:53.361057 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.361018 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2dft5"] Aug 04 15:19:53.361436 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.361380 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313f053b-53a1-4927-bc3d-de18b98fe3f3" containerName="manager" Aug 04 15:19:53.361436 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.361391 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="313f053b-53a1-4927-bc3d-de18b98fe3f3" containerName="manager" Aug 04 15:19:53.361507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.361470 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="313f053b-53a1-4927-bc3d-de18b98fe3f3" containerName="manager" Aug 04 15:19:53.364650 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.364631 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.367442 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.367413 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h67gw\"" Aug 04 15:19:53.367555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.367476 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Aug 04 15:19:53.374487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.374459 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2dft5"] Aug 04 15:19:53.406067 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.406041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:53.457298 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.457268 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2dft5"] Aug 04 15:19:53.520738 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.520707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a861f77-4706-4a87-8c08-3290f9fb86b0-config-file\") pod \"limitador-limitador-859c4d96d6-2dft5\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.520907 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.520782 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gvgl\" (UniqueName: \"kubernetes.io/projected/0a861f77-4706-4a87-8c08-3290f9fb86b0-kube-api-access-9gvgl\") pod \"limitador-limitador-859c4d96d6-2dft5\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.621684 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.621605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a861f77-4706-4a87-8c08-3290f9fb86b0-config-file\") pod \"limitador-limitador-859c4d96d6-2dft5\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.621684 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.621660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gvgl\" (UniqueName: \"kubernetes.io/projected/0a861f77-4706-4a87-8c08-3290f9fb86b0-kube-api-access-9gvgl\") pod \"limitador-limitador-859c4d96d6-2dft5\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.622224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.622205 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a861f77-4706-4a87-8c08-3290f9fb86b0-config-file\") pod \"limitador-limitador-859c4d96d6-2dft5\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.630500 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.630467 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gvgl\" (UniqueName: \"kubernetes.io/projected/0a861f77-4706-4a87-8c08-3290f9fb86b0-kube-api-access-9gvgl\") pod \"limitador-limitador-859c4d96d6-2dft5\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.681518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.681484 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:53.814140 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.814114 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2dft5"] Aug 04 15:19:53.816207 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:53.816175 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a861f77_4706_4a87_8c08_3290f9fb86b0.slice/crio-451631abe19aa4c7433d71533a329939e91da02b70435f681db62ff2329aad23 WatchSource:0}: Error finding container 451631abe19aa4c7433d71533a329939e91da02b70435f681db62ff2329aad23: Status 404 returned error can't find the container with id 451631abe19aa4c7433d71533a329939e91da02b70435f681db62ff2329aad23 Aug 04 15:19:53.920770 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:53.920688 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:54.021901 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.021868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:54.152878 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.152844 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-7jxlc"] Aug 04 15:19:54.155908 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.155888 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7jxlc" Aug 04 15:19:54.158380 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.158360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wm752\"" Aug 04 15:19:54.163747 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.163556 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7jxlc"] Aug 04 15:19:54.193010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.192988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:54.278009 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.277947 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:19:54.329561 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.329526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qgp\" (UniqueName: \"kubernetes.io/projected/c8eb2039-3556-484f-8166-bf401da21fec-kube-api-access-z7qgp\") pod \"authorino-b54df94b4-7jxlc\" (UID: \"c8eb2039-3556-484f-8166-bf401da21fec\") " pod="kuadrant-system/authorino-b54df94b4-7jxlc" Aug 04 15:19:54.430754 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.430701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qgp\" (UniqueName: \"kubernetes.io/projected/c8eb2039-3556-484f-8166-bf401da21fec-kube-api-access-z7qgp\") pod \"authorino-b54df94b4-7jxlc\" (UID: \"c8eb2039-3556-484f-8166-bf401da21fec\") " pod="kuadrant-system/authorino-b54df94b4-7jxlc" Aug 04 15:19:54.439214 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.439181 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qgp\" (UniqueName: \"kubernetes.io/projected/c8eb2039-3556-484f-8166-bf401da21fec-kube-api-access-z7qgp\") pod \"authorino-b54df94b4-7jxlc\" (UID: \"c8eb2039-3556-484f-8166-bf401da21fec\") " pod="kuadrant-system/authorino-b54df94b4-7jxlc" Aug 04 15:19:54.466913 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.466856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7jxlc" Aug 04 15:19:54.595686 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.595656 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7jxlc"] Aug 04 15:19:54.596464 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:19:54.596439 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8eb2039_3556_484f_8166_bf401da21fec.slice/crio-a39c5b8485eb4e27784194bb52044b55738dd225baed0f6209e10697beaa321f WatchSource:0}: Error finding container a39c5b8485eb4e27784194bb52044b55738dd225baed0f6209e10697beaa321f: Status 404 returned error can't find the container with id a39c5b8485eb4e27784194bb52044b55738dd225baed0f6209e10697beaa321f Aug 04 15:19:54.771616 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.771501 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7jxlc" event={"ID":"c8eb2039-3556-484f-8166-bf401da21fec","Type":"ContainerStarted","Data":"a39c5b8485eb4e27784194bb52044b55738dd225baed0f6209e10697beaa321f"} Aug 04 15:19:54.772829 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:54.772801 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" event={"ID":"0a861f77-4706-4a87-8c08-3290f9fb86b0","Type":"ContainerStarted","Data":"451631abe19aa4c7433d71533a329939e91da02b70435f681db62ff2329aad23"} Aug 04 15:19:55.782187 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:55.782141 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" event={"ID":"0a861f77-4706-4a87-8c08-3290f9fb86b0","Type":"ContainerStarted","Data":"a5c43a582138a9b1a8d4e3c9b9090e60b9a5aff297feb4db3dec18fadfb0428d"} Aug 04 15:19:55.782651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:55.782480 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:19:55.801224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:55.801168 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" podStartSLOduration=1.45988565 podStartE2EDuration="2.801148848s" totalImagesPullingTime="1.341263198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:55.798164408 +0000 UTC m=+561.152252165" watchObservedRunningTime="2026-08-04 15:19:55.801148848 +0000 UTC m=+561.155236608" Aug 04 15:19:56.788209 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:56.788167 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7jxlc" event={"ID":"c8eb2039-3556-484f-8166-bf401da21fec","Type":"ContainerStarted","Data":"8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15"} Aug 04 15:19:56.806722 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:56.806670 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-7jxlc" podStartSLOduration=0.786991016 podStartE2EDuration="2.806653623s" totalImagesPullingTime="2.019662607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:19:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:19:56.805562708 +0000 UTC m=+562.159650467" watchObservedRunningTime="2026-08-04 15:19:56.806653623 +0000 UTC m=+562.160741382" Aug 04 15:19:57.174499 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:57.174462 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7jxlc"] Aug 04 15:19:58.797452 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:58.797412 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-7jxlc" podUID="c8eb2039-3556-484f-8166-bf401da21fec" containerName="authorino" containerID="cri-o://8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15" gracePeriod=30 Aug 04 15:19:59.052014 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.051955 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7jxlc" Aug 04 15:19:59.172459 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.172427 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7qgp\" (UniqueName: \"kubernetes.io/projected/c8eb2039-3556-484f-8166-bf401da21fec-kube-api-access-z7qgp\") pod \"c8eb2039-3556-484f-8166-bf401da21fec\" (UID: \"c8eb2039-3556-484f-8166-bf401da21fec\") " Aug 04 15:19:59.174367 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.174343 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8eb2039-3556-484f-8166-bf401da21fec-kube-api-access-z7qgp" (OuterVolumeSpecName: "kube-api-access-z7qgp") pod "c8eb2039-3556-484f-8166-bf401da21fec" (UID: "c8eb2039-3556-484f-8166-bf401da21fec"). InnerVolumeSpecName "kube-api-access-z7qgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:19:59.273010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.272982 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qgp\" (UniqueName: \"kubernetes.io/projected/c8eb2039-3556-484f-8166-bf401da21fec-kube-api-access-z7qgp\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:19:59.804148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.804110 2559 generic.go:356] "Generic (PLEG): container finished" podID="c8eb2039-3556-484f-8166-bf401da21fec" containerID="8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15" exitCode=0 Aug 04 15:19:59.804516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.804155 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-7jxlc" Aug 04 15:19:59.804516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.804196 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7jxlc" event={"ID":"c8eb2039-3556-484f-8166-bf401da21fec","Type":"ContainerDied","Data":"8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15"} Aug 04 15:19:59.804516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.804240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-7jxlc" event={"ID":"c8eb2039-3556-484f-8166-bf401da21fec","Type":"ContainerDied","Data":"a39c5b8485eb4e27784194bb52044b55738dd225baed0f6209e10697beaa321f"} Aug 04 15:19:59.804516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.804259 2559 scope.go:117] "RemoveContainer" containerID="8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15" Aug 04 15:19:59.812848 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.812832 2559 scope.go:117] "RemoveContainer" containerID="8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15" Aug 04 15:19:59.813103 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:19:59.813083 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15\": container with ID starting with 8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15 not found: ID does not exist" containerID="8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15" Aug 04 15:19:59.813167 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.813110 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15"} err="failed to get container status \"8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15\": rpc error: code = NotFound desc = could not find container \"8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15\": container with ID starting with 8eb9fba89fb0708da6987c2de66452caa1268f06c1ef4526cc8362d803d1de15 not found: ID does not exist" Aug 04 15:19:59.823538 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.823514 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7jxlc"] Aug 04 15:19:59.828537 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:19:59.828519 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-7jxlc"] Aug 04 15:20:01.290322 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:01.290289 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8eb2039-3556-484f-8166-bf401da21fec" path="/var/lib/kubelet/pods/c8eb2039-3556-484f-8166-bf401da21fec/volumes" Aug 04 15:20:04.034167 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:04.034132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:04.091545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:04.091497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:06.789914 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:06.789885 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:20:08.185635 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:08.185597 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2dft5"] Aug 04 15:20:08.186030 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:08.185808 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" podUID="0a861f77-4706-4a87-8c08-3290f9fb86b0" containerName="limitador" containerID="cri-o://a5c43a582138a9b1a8d4e3c9b9090e60b9a5aff297feb4db3dec18fadfb0428d" gracePeriod=30 Aug 04 15:20:08.850307 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:08.850220 2559 generic.go:356] "Generic (PLEG): container finished" podID="0a861f77-4706-4a87-8c08-3290f9fb86b0" containerID="a5c43a582138a9b1a8d4e3c9b9090e60b9a5aff297feb4db3dec18fadfb0428d" exitCode=0 Aug 04 15:20:08.850307 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:08.850294 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" event={"ID":"0a861f77-4706-4a87-8c08-3290f9fb86b0","Type":"ContainerDied","Data":"a5c43a582138a9b1a8d4e3c9b9090e60b9a5aff297feb4db3dec18fadfb0428d"} Aug 04 15:20:09.132064 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.132037 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:20:09.156263 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.156233 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a861f77-4706-4a87-8c08-3290f9fb86b0-config-file\") pod \"0a861f77-4706-4a87-8c08-3290f9fb86b0\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " Aug 04 15:20:09.156418 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.156279 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gvgl\" (UniqueName: \"kubernetes.io/projected/0a861f77-4706-4a87-8c08-3290f9fb86b0-kube-api-access-9gvgl\") pod \"0a861f77-4706-4a87-8c08-3290f9fb86b0\" (UID: \"0a861f77-4706-4a87-8c08-3290f9fb86b0\") " Aug 04 15:20:09.156668 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.156640 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a861f77-4706-4a87-8c08-3290f9fb86b0-config-file" (OuterVolumeSpecName: "config-file") pod "0a861f77-4706-4a87-8c08-3290f9fb86b0" (UID: "0a861f77-4706-4a87-8c08-3290f9fb86b0"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:20:09.158294 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.158271 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a861f77-4706-4a87-8c08-3290f9fb86b0-kube-api-access-9gvgl" (OuterVolumeSpecName: "kube-api-access-9gvgl") pod "0a861f77-4706-4a87-8c08-3290f9fb86b0" (UID: "0a861f77-4706-4a87-8c08-3290f9fb86b0"). InnerVolumeSpecName "kube-api-access-9gvgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:20:09.256869 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.256830 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gvgl\" (UniqueName: \"kubernetes.io/projected/0a861f77-4706-4a87-8c08-3290f9fb86b0-kube-api-access-9gvgl\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:20:09.256869 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.256863 2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a861f77-4706-4a87-8c08-3290f9fb86b0-config-file\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:20:09.460687 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.460646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:09.857008 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.856933 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" Aug 04 15:20:09.857201 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.856925 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-2dft5" event={"ID":"0a861f77-4706-4a87-8c08-3290f9fb86b0","Type":"ContainerDied","Data":"451631abe19aa4c7433d71533a329939e91da02b70435f681db62ff2329aad23"} Aug 04 15:20:09.857351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.857231 2559 scope.go:117] "RemoveContainer" containerID="a5c43a582138a9b1a8d4e3c9b9090e60b9a5aff297feb4db3dec18fadfb0428d" Aug 04 15:20:09.867661 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.867565 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk"] Aug 04 15:20:09.867998 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.867984 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8eb2039-3556-484f-8166-bf401da21fec" containerName="authorino" Aug 04 15:20:09.867998 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.867998 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8eb2039-3556-484f-8166-bf401da21fec" containerName="authorino" Aug 04 15:20:09.868126 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.868020 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a861f77-4706-4a87-8c08-3290f9fb86b0" containerName="limitador" Aug 04 15:20:09.868126 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.868026 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a861f77-4706-4a87-8c08-3290f9fb86b0" containerName="limitador" Aug 04 15:20:09.868126 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.868081 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8eb2039-3556-484f-8166-bf401da21fec" containerName="authorino" Aug 04 15:20:09.868126 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.868088 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a861f77-4706-4a87-8c08-3290f9fb86b0" containerName="limitador" Aug 04 15:20:09.870776 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.870756 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:09.873116 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.873095 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Aug 04 15:20:09.873231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.873132 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Aug 04 15:20:09.873231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.873140 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Aug 04 15:20:09.873231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.873103 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xtw46\"" Aug 04 15:20:09.878975 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.878956 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2dft5"] Aug 04 15:20:09.882013 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.881994 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk"] Aug 04 15:20:09.884148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.884127 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-2dft5"] Aug 04 15:20:09.961507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.961472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0667036-68a7-44d4-ba07-2260ddd37fd7-data\") pod \"postgres-69fb4594cf-cdkpk\" (UID: \"e0667036-68a7-44d4-ba07-2260ddd37fd7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:09.961507 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:09.961511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vbf\" (UniqueName: \"kubernetes.io/projected/e0667036-68a7-44d4-ba07-2260ddd37fd7-kube-api-access-q5vbf\") pod \"postgres-69fb4594cf-cdkpk\" (UID: \"e0667036-68a7-44d4-ba07-2260ddd37fd7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:10.062853 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.062821 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0667036-68a7-44d4-ba07-2260ddd37fd7-data\") pod \"postgres-69fb4594cf-cdkpk\" (UID: \"e0667036-68a7-44d4-ba07-2260ddd37fd7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:10.063008 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.062859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vbf\" (UniqueName: \"kubernetes.io/projected/e0667036-68a7-44d4-ba07-2260ddd37fd7-kube-api-access-q5vbf\") pod \"postgres-69fb4594cf-cdkpk\" (UID: \"e0667036-68a7-44d4-ba07-2260ddd37fd7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:10.063215 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.063194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e0667036-68a7-44d4-ba07-2260ddd37fd7-data\") pod \"postgres-69fb4594cf-cdkpk\" (UID: \"e0667036-68a7-44d4-ba07-2260ddd37fd7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:10.073267 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.073241 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vbf\" (UniqueName: \"kubernetes.io/projected/e0667036-68a7-44d4-ba07-2260ddd37fd7-kube-api-access-q5vbf\") pod \"postgres-69fb4594cf-cdkpk\" (UID: \"e0667036-68a7-44d4-ba07-2260ddd37fd7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:10.151082 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.150993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:10.181713 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.181684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:10.210781 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.210751 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:10.307907 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.307884 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk"] Aug 04 15:20:10.309842 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:20:10.309814 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0667036_68a7_44d4_ba07_2260ddd37fd7.slice/crio-d7a849b458f3105c98cad49ea67d2337ba227ec1192144e7fbb47c97e75b51ca WatchSource:0}: Error finding container d7a849b458f3105c98cad49ea67d2337ba227ec1192144e7fbb47c97e75b51ca: Status 404 returned error can't find the container with id d7a849b458f3105c98cad49ea67d2337ba227ec1192144e7fbb47c97e75b51ca Aug 04 15:20:10.869542 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:10.869506 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" event={"ID":"e0667036-68a7-44d4-ba07-2260ddd37fd7","Type":"ContainerStarted","Data":"d7a849b458f3105c98cad49ea67d2337ba227ec1192144e7fbb47c97e75b51ca"} Aug 04 15:20:11.290674 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:11.290638 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a861f77-4706-4a87-8c08-3290f9fb86b0" path="/var/lib/kubelet/pods/0a861f77-4706-4a87-8c08-3290f9fb86b0/volumes" Aug 04 15:20:15.898527 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:15.898495 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" event={"ID":"e0667036-68a7-44d4-ba07-2260ddd37fd7","Type":"ContainerStarted","Data":"53a437e05259b70122ddd5ca08163ac181ab1c258a26202cac502e236bca6f3b"} Aug 04 15:20:15.899004 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:15.898616 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:15.915980 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:15.915926 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" podStartSLOduration=1.973655848 podStartE2EDuration="6.915912295s" totalImagesPullingTime="4.942256447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:20:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:20:15.914804373 +0000 UTC m=+581.268892133" watchObservedRunningTime="2026-08-04 15:20:15.915912295 +0000 UTC m=+581.270000054" Aug 04 15:20:17.703634 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:17.703603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:21.931935 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:21.931898 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cdkpk" Aug 04 15:20:22.240232 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.240192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:22.473315 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.473279 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-95hx6"] Aug 04 15:20:22.479351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.479290 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" Aug 04 15:20:22.481949 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.481925 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wm752\"" Aug 04 15:20:22.484965 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.484942 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-95hx6"] Aug 04 15:20:22.571628 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.571519 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84l9\" (UniqueName: \"kubernetes.io/projected/3dbdee2c-f4f3-4af9-9c6d-2d5271576a46-kube-api-access-d84l9\") pod \"authorino-5dc7ccb5f-95hx6\" (UID: \"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46\") " pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" Aug 04 15:20:22.672831 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.672796 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84l9\" (UniqueName: \"kubernetes.io/projected/3dbdee2c-f4f3-4af9-9c6d-2d5271576a46-kube-api-access-d84l9\") pod \"authorino-5dc7ccb5f-95hx6\" (UID: \"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46\") " pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" Aug 04 15:20:22.681686 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.681658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84l9\" (UniqueName: \"kubernetes.io/projected/3dbdee2c-f4f3-4af9-9c6d-2d5271576a46-kube-api-access-d84l9\") pod \"authorino-5dc7ccb5f-95hx6\" (UID: \"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46\") " pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" Aug 04 15:20:22.719181 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.719150 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-95hx6"] Aug 04 15:20:22.719409 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.719397 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" Aug 04 15:20:22.747126 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.747095 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d78487f86-9nz2f"] Aug 04 15:20:22.753987 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.753961 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d78487f86-9nz2f" Aug 04 15:20:22.758668 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.758641 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d78487f86-9nz2f"] Aug 04 15:20:22.839634 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.839551 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d78487f86-9nz2f"] Aug 04 15:20:22.840028 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:20:22.839994 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-c8z2x], unattached volumes=[], failed to process volumes=[kube-api-access-c8z2x]: context canceled" pod="kuadrant-system/authorino-7d78487f86-9nz2f" podUID="51fb5531-516f-4673-9dd9-cd3b173a4529" Aug 04 15:20:22.865727 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.865544 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-95hx6"] Aug 04 15:20:22.868183 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:20:22.868156 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dbdee2c_f4f3_4af9_9c6d_2d5271576a46.slice/crio-c14d5158d683386bdcf044dee9dcac88f266fce148975f21ae50c22dd377b871 WatchSource:0}: Error finding container c14d5158d683386bdcf044dee9dcac88f266fce148975f21ae50c22dd377b871: Status 404 returned error can't find the container with id c14d5158d683386bdcf044dee9dcac88f266fce148975f21ae50c22dd377b871 Aug 04 15:20:22.874490 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.874456 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8z2x\" (UniqueName: \"kubernetes.io/projected/51fb5531-516f-4673-9dd9-cd3b173a4529-kube-api-access-c8z2x\") pod \"authorino-7d78487f86-9nz2f\" (UID: \"51fb5531-516f-4673-9dd9-cd3b173a4529\") " pod="kuadrant-system/authorino-7d78487f86-9nz2f" Aug 04 15:20:22.932318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.932279 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" event={"ID":"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46","Type":"ContainerStarted","Data":"c14d5158d683386bdcf044dee9dcac88f266fce148975f21ae50c22dd377b871"} Aug 04 15:20:22.932318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.932298 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d78487f86-9nz2f" Aug 04 15:20:22.944756 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.944728 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d78487f86-9nz2f" Aug 04 15:20:22.975092 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.975057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8z2x\" (UniqueName: \"kubernetes.io/projected/51fb5531-516f-4673-9dd9-cd3b173a4529-kube-api-access-c8z2x\") pod \"authorino-7d78487f86-9nz2f\" (UID: \"51fb5531-516f-4673-9dd9-cd3b173a4529\") " pod="kuadrant-system/authorino-7d78487f86-9nz2f" Aug 04 15:20:22.984117 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:22.984086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8z2x\" (UniqueName: \"kubernetes.io/projected/51fb5531-516f-4673-9dd9-cd3b173a4529-kube-api-access-c8z2x\") pod \"authorino-7d78487f86-9nz2f\" (UID: \"51fb5531-516f-4673-9dd9-cd3b173a4529\") " pod="kuadrant-system/authorino-7d78487f86-9nz2f" Aug 04 15:20:23.075892 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:23.075855 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8z2x\" (UniqueName: \"kubernetes.io/projected/51fb5531-516f-4673-9dd9-cd3b173a4529-kube-api-access-c8z2x\") pod \"51fb5531-516f-4673-9dd9-cd3b173a4529\" (UID: \"51fb5531-516f-4673-9dd9-cd3b173a4529\") " Aug 04 15:20:23.078000 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:23.077963 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fb5531-516f-4673-9dd9-cd3b173a4529-kube-api-access-c8z2x" (OuterVolumeSpecName: "kube-api-access-c8z2x") pod "51fb5531-516f-4673-9dd9-cd3b173a4529" (UID: "51fb5531-516f-4673-9dd9-cd3b173a4529"). InnerVolumeSpecName "kube-api-access-c8z2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:20:23.084523 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:23.084487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:23.177246 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:23.177166 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8z2x\" (UniqueName: \"kubernetes.io/projected/51fb5531-516f-4673-9dd9-cd3b173a4529-kube-api-access-c8z2x\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:20:23.938325 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:23.938236 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d78487f86-9nz2f" Aug 04 15:20:23.969307 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:23.969278 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d78487f86-9nz2f"] Aug 04 15:20:23.971656 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:23.971631 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d78487f86-9nz2f"] Aug 04 15:20:25.288987 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:25.288954 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fb5531-516f-4673-9dd9-cd3b173a4529" path="/var/lib/kubelet/pods/51fb5531-516f-4673-9dd9-cd3b173a4529/volumes" Aug 04 15:20:25.948652 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:25.948615 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" event={"ID":"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46","Type":"ContainerStarted","Data":"a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205"} Aug 04 15:20:25.948652 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:25.948640 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" podUID="3dbdee2c-f4f3-4af9-9c6d-2d5271576a46" containerName="authorino" containerID="cri-o://a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205" gracePeriod=30 Aug 04 15:20:25.965686 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:25.965646 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" podStartSLOduration=1.6712444880000001 podStartE2EDuration="3.96563243s" totalImagesPullingTime="2.294387942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:20:25.96362984 +0000 UTC m=+591.317717602" watchObservedRunningTime="2026-08-04 15:20:25.96563243 +0000 UTC m=+591.319720189" Aug 04 15:20:26.187754 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.187731 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" Aug 04 15:20:26.300621 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.300489 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d84l9\" (UniqueName: \"kubernetes.io/projected/3dbdee2c-f4f3-4af9-9c6d-2d5271576a46-kube-api-access-d84l9\") pod \"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46\" (UID: \"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46\") " Aug 04 15:20:26.302555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.302529 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbdee2c-f4f3-4af9-9c6d-2d5271576a46-kube-api-access-d84l9" (OuterVolumeSpecName: "kube-api-access-d84l9") pod "3dbdee2c-f4f3-4af9-9c6d-2d5271576a46" (UID: "3dbdee2c-f4f3-4af9-9c6d-2d5271576a46"). InnerVolumeSpecName "kube-api-access-d84l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:20:26.401926 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.401892 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84l9\" (UniqueName: \"kubernetes.io/projected/3dbdee2c-f4f3-4af9-9c6d-2d5271576a46-kube-api-access-d84l9\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:20:26.954907 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.954867 2559 generic.go:356] "Generic (PLEG): container finished" podID="3dbdee2c-f4f3-4af9-9c6d-2d5271576a46" containerID="a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205" exitCode=0 Aug 04 15:20:26.955098 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.954909 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" Aug 04 15:20:26.955098 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.954924 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" event={"ID":"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46","Type":"ContainerDied","Data":"a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205"} Aug 04 15:20:26.955098 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.954960 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-95hx6" event={"ID":"3dbdee2c-f4f3-4af9-9c6d-2d5271576a46","Type":"ContainerDied","Data":"c14d5158d683386bdcf044dee9dcac88f266fce148975f21ae50c22dd377b871"} Aug 04 15:20:26.955098 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.954978 2559 scope.go:117] "RemoveContainer" containerID="a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205" Aug 04 15:20:26.964243 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.964203 2559 scope.go:117] "RemoveContainer" containerID="a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205" Aug 04 15:20:26.964482 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:20:26.964464 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205\": container with ID starting with a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205 not found: ID does not exist" containerID="a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205" Aug 04 15:20:26.964533 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.964490 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205"} err="failed to get container status \"a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205\": rpc error: code = NotFound desc = could not find container \"a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205\": container with ID starting with a52e0f4c441ee4eca7b4ee0f43b76598037ff0d692b38b4f23e5aad26fed5205 not found: ID does not exist" Aug 04 15:20:26.975876 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.975855 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-95hx6"] Aug 04 15:20:26.979402 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:26.979381 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-95hx6"] Aug 04 15:20:27.287973 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:27.287892 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dbdee2c-f4f3-4af9-9c6d-2d5271576a46" path="/var/lib/kubelet/pods/3dbdee2c-f4f3-4af9-9c6d-2d5271576a46/volumes" Aug 04 15:20:29.030790 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.030757 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:29.052969 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.052935 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5b969d7d64-d75vz"] Aug 04 15:20:29.053341 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.053328 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dbdee2c-f4f3-4af9-9c6d-2d5271576a46" containerName="authorino" Aug 04 15:20:29.053397 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.053343 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dbdee2c-f4f3-4af9-9c6d-2d5271576a46" containerName="authorino" Aug 04 15:20:29.053436 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.053400 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dbdee2c-f4f3-4af9-9c6d-2d5271576a46" containerName="authorino" Aug 04 15:20:29.056398 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.056381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.058989 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.058963 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r9z9v\"" Aug 04 15:20:29.059099 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.059008 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 15:20:29.059099 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.058975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 15:20:29.066795 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.066772 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b969d7d64-d75vz"] Aug 04 15:20:29.112346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.112309 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:29.127726 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.127696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert\") pod \"maas-controller-5b969d7d64-d75vz\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.127859 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.127743 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8shk\" (UniqueName: \"kubernetes.io/projected/dc1212dc-3638-4615-8f47-2d43f804b832-kube-api-access-p8shk\") pod \"maas-controller-5b969d7d64-d75vz\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.173978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.173934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:29.217606 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.217564 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78bcb9f478-nwmsb"] Aug 04 15:20:29.220637 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.220619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.228458 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.228433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8shk\" (UniqueName: \"kubernetes.io/projected/dc1212dc-3638-4615-8f47-2d43f804b832-kube-api-access-p8shk\") pod \"maas-controller-5b969d7d64-d75vz\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.228600 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.228516 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert\") pod \"maas-controller-5b969d7d64-d75vz\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.228678 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:20:29.228618 2559 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 15:20:29.228844 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:20:29.228691 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert podName:dc1212dc-3638-4615-8f47-2d43f804b832 nodeName:}" failed. No retries permitted until 2026-08-04 15:20:29.728669759 +0000 UTC m=+595.082757500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert") pod "maas-controller-5b969d7d64-d75vz" (UID: "dc1212dc-3638-4615-8f47-2d43f804b832") : secret "maas-controller-webhook-cert" not found Aug 04 15:20:29.232855 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.232830 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-nwmsb"] Aug 04 15:20:29.240658 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.240632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8shk\" (UniqueName: \"kubernetes.io/projected/dc1212dc-3638-4615-8f47-2d43f804b832-kube-api-access-p8shk\") pod \"maas-controller-5b969d7d64-d75vz\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.329847 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.329765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert\") pod \"maas-controller-78bcb9f478-nwmsb\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.329993 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.329863 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ttd\" (UniqueName: \"kubernetes.io/projected/d0d81a76-367a-4cda-9983-8667444d3c28-kube-api-access-t7ttd\") pod \"maas-controller-78bcb9f478-nwmsb\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.430297 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.430262 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ttd\" (UniqueName: \"kubernetes.io/projected/d0d81a76-367a-4cda-9983-8667444d3c28-kube-api-access-t7ttd\") pod \"maas-controller-78bcb9f478-nwmsb\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.430615 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.430324 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert\") pod \"maas-controller-78bcb9f478-nwmsb\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.430615 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:20:29.430416 2559 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Aug 04 15:20:29.430615 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:20:29.430480 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert podName:d0d81a76-367a-4cda-9983-8667444d3c28 nodeName:}" failed. No retries permitted until 2026-08-04 15:20:29.930461677 +0000 UTC m=+595.284549416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert") pod "maas-controller-78bcb9f478-nwmsb" (UID: "d0d81a76-367a-4cda-9983-8667444d3c28") : secret "maas-controller-webhook-cert" not found Aug 04 15:20:29.448036 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.448003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ttd\" (UniqueName: \"kubernetes.io/projected/d0d81a76-367a-4cda-9983-8667444d3c28-kube-api-access-t7ttd\") pod \"maas-controller-78bcb9f478-nwmsb\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.619214 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.619132 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:29.731979 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.731947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert\") pod \"maas-controller-5b969d7d64-d75vz\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.734231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.734212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert\") pod \"maas-controller-5b969d7d64-d75vz\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:29.933797 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.933716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert\") pod \"maas-controller-78bcb9f478-nwmsb\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.936056 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.936037 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert\") pod \"maas-controller-78bcb9f478-nwmsb\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:29.970274 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:29.970248 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:30.103539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:30.103512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5b969d7d64-d75vz"] Aug 04 15:20:30.105555 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:20:30.105528 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1212dc_3638_4615_8f47_2d43f804b832.slice/crio-5b62d6422e304ee698c592a17ac89dd0820f33165ba2bb2f99d762edfda95ba3 WatchSource:0}: Error finding container 5b62d6422e304ee698c592a17ac89dd0820f33165ba2bb2f99d762edfda95ba3: Status 404 returned error can't find the container with id 5b62d6422e304ee698c592a17ac89dd0820f33165ba2bb2f99d762edfda95ba3 Aug 04 15:20:30.132759 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:30.132735 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:30.258871 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:30.258849 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-nwmsb"] Aug 04 15:20:30.260764 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:20:30.260737 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d81a76_367a_4cda_9983_8667444d3c28.slice/crio-f4c3f43accac06994b1e82c0276c2d0ee7c7a6b76052e987067fb2797674ef5f WatchSource:0}: Error finding container f4c3f43accac06994b1e82c0276c2d0ee7c7a6b76052e987067fb2797674ef5f: Status 404 returned error can't find the container with id f4c3f43accac06994b1e82c0276c2d0ee7c7a6b76052e987067fb2797674ef5f Aug 04 15:20:30.979479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:30.979435 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b969d7d64-d75vz" event={"ID":"dc1212dc-3638-4615-8f47-2d43f804b832","Type":"ContainerStarted","Data":"5b62d6422e304ee698c592a17ac89dd0820f33165ba2bb2f99d762edfda95ba3"} Aug 04 15:20:30.981105 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:30.981075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" event={"ID":"d0d81a76-367a-4cda-9983-8667444d3c28","Type":"ContainerStarted","Data":"f4c3f43accac06994b1e82c0276c2d0ee7c7a6b76052e987067fb2797674ef5f"} Aug 04 15:20:32.805998 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:32.805956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:32.906730 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:32.906633 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:20:32.992545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:32.992499 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" event={"ID":"d0d81a76-367a-4cda-9983-8667444d3c28","Type":"ContainerStarted","Data":"a910d3e9f77a393f9d82de6b7de54d3bb578be9a00e72e0c21a727db5c46bd3c"} Aug 04 15:20:32.992784 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:32.992705 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:32.993999 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:32.993976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b969d7d64-d75vz" event={"ID":"dc1212dc-3638-4615-8f47-2d43f804b832","Type":"ContainerStarted","Data":"1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb"} Aug 04 15:20:32.994146 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:32.994015 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:33.012110 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:33.012065 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" podStartSLOduration=1.632667595 podStartE2EDuration="4.01204992s" totalImagesPullingTime="2.379382325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:20:33.010669163 +0000 UTC m=+598.364756923" watchObservedRunningTime="2026-08-04 15:20:33.01204992 +0000 UTC m=+598.366137657" Aug 04 15:20:33.032273 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:33.032229 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5b969d7d64-d75vz" podStartSLOduration=1.505093929 podStartE2EDuration="4.032214327s" totalImagesPullingTime="2.527120398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:20:33.030597402 +0000 UTC m=+598.384685162" watchObservedRunningTime="2026-08-04 15:20:33.032214327 +0000 UTC m=+598.386302086" Aug 04 15:20:35.090349 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:35.090313 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:20:35.090816 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:35.090671 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:20:35.095240 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:35.095214 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:20:35.095497 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:35.095477 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:20:44.001536 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.001505 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:44.002187 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.001596 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:20:44.065224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.065184 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b969d7d64-d75vz"] Aug 04 15:20:44.065532 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.065472 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5b969d7d64-d75vz" podUID="dc1212dc-3638-4615-8f47-2d43f804b832" containerName="manager" containerID="cri-o://1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb" gracePeriod=30 Aug 04 15:20:44.316272 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.316245 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:44.458439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.458400 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert\") pod \"dc1212dc-3638-4615-8f47-2d43f804b832\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " Aug 04 15:20:44.458439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.458443 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8shk\" (UniqueName: \"kubernetes.io/projected/dc1212dc-3638-4615-8f47-2d43f804b832-kube-api-access-p8shk\") pod \"dc1212dc-3638-4615-8f47-2d43f804b832\" (UID: \"dc1212dc-3638-4615-8f47-2d43f804b832\") " Aug 04 15:20:44.460595 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.460546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dc1212dc-3638-4615-8f47-2d43f804b832" (UID: "dc1212dc-3638-4615-8f47-2d43f804b832"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:20:44.460595 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.460551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1212dc-3638-4615-8f47-2d43f804b832-kube-api-access-p8shk" (OuterVolumeSpecName: "kube-api-access-p8shk") pod "dc1212dc-3638-4615-8f47-2d43f804b832" (UID: "dc1212dc-3638-4615-8f47-2d43f804b832"). InnerVolumeSpecName "kube-api-access-p8shk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:20:44.559135 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.559063 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc1212dc-3638-4615-8f47-2d43f804b832-webhook-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:20:44.559135 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:44.559090 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8shk\" (UniqueName: \"kubernetes.io/projected/dc1212dc-3638-4615-8f47-2d43f804b832-kube-api-access-p8shk\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:20:45.052744 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.052706 2559 generic.go:356] "Generic (PLEG): container finished" podID="dc1212dc-3638-4615-8f47-2d43f804b832" containerID="1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb" exitCode=0 Aug 04 15:20:45.053142 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.052760 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5b969d7d64-d75vz" Aug 04 15:20:45.053142 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.052792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b969d7d64-d75vz" event={"ID":"dc1212dc-3638-4615-8f47-2d43f804b832","Type":"ContainerDied","Data":"1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb"} Aug 04 15:20:45.053142 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.052865 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5b969d7d64-d75vz" event={"ID":"dc1212dc-3638-4615-8f47-2d43f804b832","Type":"ContainerDied","Data":"5b62d6422e304ee698c592a17ac89dd0820f33165ba2bb2f99d762edfda95ba3"} Aug 04 15:20:45.053142 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.052887 2559 scope.go:117] "RemoveContainer" containerID="1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb" Aug 04 15:20:45.062094 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.062079 2559 scope.go:117] "RemoveContainer" containerID="1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb" Aug 04 15:20:45.062344 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:20:45.062326 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb\": container with ID starting with 1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb not found: ID does not exist" containerID="1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb" Aug 04 15:20:45.062407 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.062357 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb"} err="failed to get container status \"1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb\": rpc error: code = NotFound desc = could not find container \"1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb\": container with ID starting with 1369bfa4835b1fd9cd38df337677837cf14c28c66475919a3da7b08a952ad3cb not found: ID does not exist" Aug 04 15:20:45.074669 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.074646 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5b969d7d64-d75vz"] Aug 04 15:20:45.079134 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.079115 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5b969d7d64-d75vz"] Aug 04 15:20:45.287525 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:20:45.287494 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc1212dc-3638-4615-8f47-2d43f804b832" path="/var/lib/kubelet/pods/dc1212dc-3638-4615-8f47-2d43f804b832/volumes" Aug 04 15:21:03.761346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:03.761267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:05.281854 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.281800 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:05.447059 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.447023 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:05.496925 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.496893 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj"] Aug 04 15:21:05.497304 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.497290 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc1212dc-3638-4615-8f47-2d43f804b832" containerName="manager" Aug 04 15:21:05.497355 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.497306 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc1212dc-3638-4615-8f47-2d43f804b832" containerName="manager" Aug 04 15:21:05.497391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.497382 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc1212dc-3638-4615-8f47-2d43f804b832" containerName="manager" Aug 04 15:21:05.500329 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.500310 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.503850 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.503828 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Aug 04 15:21:05.503850 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.503841 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nrqs6\"" Aug 04 15:21:05.512367 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.512347 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj"] Aug 04 15:21:05.633597 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.633503 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f65295-fdcd-47a2-a3dc-78809c2219c0-openshift-service-ca\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.633597 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.633538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df6n6\" (UniqueName: \"kubernetes.io/projected/28f65295-fdcd-47a2-a3dc-78809c2219c0-kube-api-access-df6n6\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.633780 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.633661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f65295-fdcd-47a2-a3dc-78809c2219c0-maas-api-tls\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.734294 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.734255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f65295-fdcd-47a2-a3dc-78809c2219c0-maas-api-tls\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.734479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.734305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f65295-fdcd-47a2-a3dc-78809c2219c0-openshift-service-ca\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.734479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.734322 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df6n6\" (UniqueName: \"kubernetes.io/projected/28f65295-fdcd-47a2-a3dc-78809c2219c0-kube-api-access-df6n6\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.734944 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.734924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f65295-fdcd-47a2-a3dc-78809c2219c0-openshift-service-ca\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.736950 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.736927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f65295-fdcd-47a2-a3dc-78809c2219c0-maas-api-tls\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.742385 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.742363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df6n6\" (UniqueName: \"kubernetes.io/projected/28f65295-fdcd-47a2-a3dc-78809c2219c0-kube-api-access-df6n6\") pod \"maas-api-5bf88d8cb-9mzrj\" (UID: \"28f65295-fdcd-47a2-a3dc-78809c2219c0\") " pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.773968 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.773929 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:05.811028 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.810999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:05.919437 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.919386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:05.954453 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:05.954424 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj"] Aug 04 15:21:05.956607 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:21:05.956547 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f65295_fdcd_47a2_a3dc_78809c2219c0.slice/crio-77db37e6a3534bc6911405d2eae819fd4bba2ca29bc44c76ff74c05318f727c2 WatchSource:0}: Error finding container 77db37e6a3534bc6911405d2eae819fd4bba2ca29bc44c76ff74c05318f727c2: Status 404 returned error can't find the container with id 77db37e6a3534bc6911405d2eae819fd4bba2ca29bc44c76ff74c05318f727c2 Aug 04 15:21:06.017215 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.017182 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:06.068592 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.068542 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pzfkj"] Aug 04 15:21:06.073154 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.073130 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.075532 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.075511 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6hvbs\"" Aug 04 15:21:06.075701 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.075553 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Aug 04 15:21:06.087054 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.087032 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pzfkj"] Aug 04 15:21:06.156619 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.156532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" event={"ID":"28f65295-fdcd-47a2-a3dc-78809c2219c0","Type":"ContainerStarted","Data":"77db37e6a3534bc6911405d2eae819fd4bba2ca29bc44c76ff74c05318f727c2"} Aug 04 15:21:06.239368 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.239332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3494aa11-cbd1-4c2f-bae2-b565ef4d1f80-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pzfkj\" (UID: \"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.239531 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.239384 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xx2g\" (UniqueName: \"kubernetes.io/projected/3494aa11-cbd1-4c2f-bae2-b565ef4d1f80-kube-api-access-9xx2g\") pod \"payload-processing-5cc6cd444b-pzfkj\" (UID: \"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.340847 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.340816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3494aa11-cbd1-4c2f-bae2-b565ef4d1f80-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pzfkj\" (UID: \"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.341313 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.340883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xx2g\" (UniqueName: \"kubernetes.io/projected/3494aa11-cbd1-4c2f-bae2-b565ef4d1f80-kube-api-access-9xx2g\") pod \"payload-processing-5cc6cd444b-pzfkj\" (UID: \"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.341537 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.341515 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3494aa11-cbd1-4c2f-bae2-b565ef4d1f80-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-pzfkj\" (UID: \"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.348498 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.348474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xx2g\" (UniqueName: \"kubernetes.io/projected/3494aa11-cbd1-4c2f-bae2-b565ef4d1f80-kube-api-access-9xx2g\") pod \"payload-processing-5cc6cd444b-pzfkj\" (UID: \"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80\") " pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.383118 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.383085 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:06.516884 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:06.516752 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-pzfkj"] Aug 04 15:21:06.518962 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:21:06.518935 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3494aa11_cbd1_4c2f_bae2_b565ef4d1f80.slice/crio-f661fcf66806152e7214288509f38a469c5685b077ab0b8966f7dcd287488812 WatchSource:0}: Error finding container f661fcf66806152e7214288509f38a469c5685b077ab0b8966f7dcd287488812: Status 404 returned error can't find the container with id f661fcf66806152e7214288509f38a469c5685b077ab0b8966f7dcd287488812 Aug 04 15:21:07.163358 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:07.163320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" event={"ID":"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80","Type":"ContainerStarted","Data":"f661fcf66806152e7214288509f38a469c5685b077ab0b8966f7dcd287488812"} Aug 04 15:21:08.170681 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:08.170590 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" event={"ID":"28f65295-fdcd-47a2-a3dc-78809c2219c0","Type":"ContainerStarted","Data":"ebdf7461ca1cd148224698a4b6af0ac5e51f036ead325f3e0fc9f8e256671bca"} Aug 04 15:21:08.170681 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:08.170661 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:08.188052 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:08.187992 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" podStartSLOduration=1.367839591 podStartE2EDuration="3.187973298s" totalImagesPullingTime="1.820133707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:21:08.187368056 +0000 UTC m=+633.541455819" watchObservedRunningTime="2026-08-04 15:21:08.187973298 +0000 UTC m=+633.542061059" Aug 04 15:21:09.176660 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:09.176553 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" event={"ID":"3494aa11-cbd1-4c2f-bae2-b565ef4d1f80","Type":"ContainerStarted","Data":"a33345f2489682bbaece6fce42f109858646b74c391cadc54457b247b1f9378e"} Aug 04 15:21:09.194279 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:09.194231 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" podStartSLOduration=0.815473914 podStartE2EDuration="3.194216446s" totalImagesPullingTime="2.378742532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:21:09.192435316 +0000 UTC m=+634.546523077" watchObservedRunningTime="2026-08-04 15:21:09.194216446 +0000 UTC m=+634.548304204" Aug 04 15:21:10.078030 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:10.077988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:10.180859 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:10.180831 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:14.182092 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:14.182055 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bf88d8cb-9mzrj" Aug 04 15:21:15.847634 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:15.847590 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:15.947345 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:15.947311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:21.188127 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:21.188091 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-pzfkj" Aug 04 15:21:25.893689 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.893658 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm"] Aug 04 15:21:25.896832 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.896814 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:25.900090 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.900066 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Aug 04 15:21:25.900227 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.900096 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n5m5q\"" Aug 04 15:21:25.900227 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.900074 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Aug 04 15:21:25.900227 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.900072 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Aug 04 15:21:25.907217 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.907197 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm"] Aug 04 15:21:25.921303 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:25.921270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:26.008521 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.008484 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.008713 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.008538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.008713 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.008645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.008713 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.008677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z996m\" (UniqueName: \"kubernetes.io/projected/5aed4231-9239-406f-8caa-9c897350e55c-kube-api-access-z996m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.008713 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.008710 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.008922 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.008791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed4231-9239-406f-8caa-9c897350e55c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.109494 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.109494 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109503 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z996m\" (UniqueName: \"kubernetes.io/projected/5aed4231-9239-406f-8caa-9c897350e55c-kube-api-access-z996m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.109828 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.109828 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed4231-9239-406f-8caa-9c897350e55c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.109828 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109647 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.109828 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109710 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.110058 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.110058 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.109975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.110156 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.110099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.111890 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.111866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5aed4231-9239-406f-8caa-9c897350e55c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.112078 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.112060 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5aed4231-9239-406f-8caa-9c897350e55c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.120767 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.120743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z996m\" (UniqueName: \"kubernetes.io/projected/5aed4231-9239-406f-8caa-9c897350e55c-kube-api-access-z996m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm\" (UID: \"5aed4231-9239-406f-8caa-9c897350e55c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.207832 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.207800 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:26.343485 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:26.343460 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm"] Aug 04 15:21:26.345258 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:21:26.345230 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aed4231_9239_406f_8caa_9c897350e55c.slice/crio-119d074e35d22fe783e36098f279b1144b0ed1f222e9e328c442cc409c22eeaf WatchSource:0}: Error finding container 119d074e35d22fe783e36098f279b1144b0ed1f222e9e328c442cc409c22eeaf: Status 404 returned error can't find the container with id 119d074e35d22fe783e36098f279b1144b0ed1f222e9e328c442cc409c22eeaf Aug 04 15:21:27.267675 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:27.267637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" event={"ID":"5aed4231-9239-406f-8caa-9c897350e55c","Type":"ContainerStarted","Data":"119d074e35d22fe783e36098f279b1144b0ed1f222e9e328c442cc409c22eeaf"} Aug 04 15:21:28.273524 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:28.273482 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" event={"ID":"5aed4231-9239-406f-8caa-9c897350e55c","Type":"ContainerStarted","Data":"447a41f01892ec0143e2d2d061e9365c01a791a672bd38ee63a079321349f618"} Aug 04 15:21:28.292834 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:28.292784 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" podStartSLOduration=2.201317848 podStartE2EDuration="3.292768688s" totalImagesPullingTime="1.09145084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:21:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:21:28.291773256 +0000 UTC m=+653.645861016" watchObservedRunningTime="2026-08-04 15:21:28.292768688 +0000 UTC m=+653.646856446" Aug 04 15:21:32.851748 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:32.851708 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:21:36.208114 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:36.208075 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:36.208617 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:36.208169 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:36.220881 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:36.220857 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:36.323934 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:36.323905 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm" Aug 04 15:21:53.930428 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:21:53.930390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:22:12.716935 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.716899 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g"] Aug 04 15:22:12.719897 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.719873 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.728766 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.728742 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Aug 04 15:22:12.735262 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.735244 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g"] Aug 04 15:22:12.776510 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.776479 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:22:12.805647 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.805614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.805647 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.805642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.805806 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.805660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtd7m\" (UniqueName: \"kubernetes.io/projected/21092b43-962a-4beb-8b47-1c16bf3ff516-kube-api-access-dtd7m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.805806 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.805719 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.805806 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.805758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.805806 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.805782 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21092b43-962a-4beb-8b47-1c16bf3ff516-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.906647 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.906612 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21092b43-962a-4beb-8b47-1c16bf3ff516-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.906825 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.906774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.906825 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.906809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.906947 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.906828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtd7m\" (UniqueName: \"kubernetes.io/projected/21092b43-962a-4beb-8b47-1c16bf3ff516-kube-api-access-dtd7m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.906947 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.906865 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.906947 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.906890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.907203 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.907174 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.907318 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.907289 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.907431 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.907318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.909002 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.908982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21092b43-962a-4beb-8b47-1c16bf3ff516-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.909295 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.909276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21092b43-962a-4beb-8b47-1c16bf3ff516-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:12.915335 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:12.915317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtd7m\" (UniqueName: \"kubernetes.io/projected/21092b43-962a-4beb-8b47-1c16bf3ff516-kube-api-access-dtd7m\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g\" (UID: \"21092b43-962a-4beb-8b47-1c16bf3ff516\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:13.029852 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:13.029769 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:13.163407 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:13.163381 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g"] Aug 04 15:22:13.165785 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:22:13.165756 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21092b43_962a_4beb_8b47_1c16bf3ff516.slice/crio-76edfbf689b97ae293410fa913564fc998efbe8d5fdd8235bd0d308c2caabffc WatchSource:0}: Error finding container 76edfbf689b97ae293410fa913564fc998efbe8d5fdd8235bd0d308c2caabffc: Status 404 returned error can't find the container with id 76edfbf689b97ae293410fa913564fc998efbe8d5fdd8235bd0d308c2caabffc Aug 04 15:22:13.167673 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:13.167648 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:22:13.486864 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:13.486832 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" event={"ID":"21092b43-962a-4beb-8b47-1c16bf3ff516","Type":"ContainerStarted","Data":"8f6ecba37f2d023ecb9ff51c8b60f25458d84cb5b55459ec6cb1b3c9637a2ddb"} Aug 04 15:22:13.487076 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:13.486881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" event={"ID":"21092b43-962a-4beb-8b47-1c16bf3ff516","Type":"ContainerStarted","Data":"76edfbf689b97ae293410fa913564fc998efbe8d5fdd8235bd0d308c2caabffc"} Aug 04 15:22:13.505877 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:13.505836 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" podStartSLOduration=1.316722241 podStartE2EDuration="1.505820451s" totalImagesPullingTime="189.09821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:22:13.504831018 +0000 UTC m=+698.858918777" watchObservedRunningTime="2026-08-04 15:22:13.505820451 +0000 UTC m=+698.859908209" Aug 04 15:22:23.030881 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:23.030819 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:23.031386 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:23.030905 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:23.043496 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:23.043470 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:23.546392 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:23.546358 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g" Aug 04 15:22:28.152983 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:28.152904 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:22:28.254205 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:28.254165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:22:31.798077 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:31.798031 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:22:47.878964 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:47.878924 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:22:49.381850 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:49.381807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:22:57.234180 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:22:57.234139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:23:06.459448 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:06.459403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:23:57.378733 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.378642 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b59d4dc5c-b5fkg"] Aug 04 15:23:57.381594 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.381559 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.385911 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.385889 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-wm752\"" Aug 04 15:23:57.386795 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.386779 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Aug 04 15:23:57.392251 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.392224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b59d4dc5c-b5fkg"] Aug 04 15:23:57.422975 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.422947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77ks\" (UniqueName: \"kubernetes.io/projected/9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a-kube-api-access-j77ks\") pod \"authorino-b59d4dc5c-b5fkg\" (UID: \"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a\") " pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.423137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.423018 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a-tls-cert\") pod \"authorino-b59d4dc5c-b5fkg\" (UID: \"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a\") " pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.523730 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.523693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a-tls-cert\") pod \"authorino-b59d4dc5c-b5fkg\" (UID: \"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a\") " pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.523925 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.523755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77ks\" (UniqueName: \"kubernetes.io/projected/9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a-kube-api-access-j77ks\") pod \"authorino-b59d4dc5c-b5fkg\" (UID: \"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a\") " pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.526234 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.526209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a-tls-cert\") pod \"authorino-b59d4dc5c-b5fkg\" (UID: \"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a\") " pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.532210 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.532187 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77ks\" (UniqueName: \"kubernetes.io/projected/9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a-kube-api-access-j77ks\") pod \"authorino-b59d4dc5c-b5fkg\" (UID: \"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a\") " pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.691337 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.691255 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" Aug 04 15:23:57.824833 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:57.824804 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b59d4dc5c-b5fkg"] Aug 04 15:23:57.828080 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:23:57.828055 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d6ff147_a6f7_4d4e_9e3e_3e9c814e885a.slice/crio-07f201017095657db0310080bffd1ae1b7347d33787df890a4f7162692974157 WatchSource:0}: Error finding container 07f201017095657db0310080bffd1ae1b7347d33787df890a4f7162692974157: Status 404 returned error can't find the container with id 07f201017095657db0310080bffd1ae1b7347d33787df890a4f7162692974157 Aug 04 15:23:58.001120 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:23:58.001087 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" event={"ID":"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a","Type":"ContainerStarted","Data":"07f201017095657db0310080bffd1ae1b7347d33787df890a4f7162692974157"} Aug 04 15:24:07.049596 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:07.049536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" event={"ID":"9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a","Type":"ContainerStarted","Data":"050bd789f8ef2af63add444fb3de4562b197666969280f59f676db40ae217c4a"} Aug 04 15:24:07.072035 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:07.071980 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b59d4dc5c-b5fkg" podStartSLOduration=1.3946695949999999 podStartE2EDuration="10.071964608s" totalImagesPullingTime="8.677295013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:23:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:24:07.069215284 +0000 UTC m=+812.423303043" watchObservedRunningTime="2026-08-04 15:24:07.071964608 +0000 UTC m=+812.426052380" Aug 04 15:24:07.623750 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:07.623712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:24:07.722782 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:07.722747 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:24:08.725351 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:08.725313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:24:08.827685 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:08.827654 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:24:55.058548 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.058508 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-nwmsb"] Aug 04 15:24:55.059123 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.058783 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" podUID="d0d81a76-367a-4cda-9983-8667444d3c28" containerName="manager" containerID="cri-o://a910d3e9f77a393f9d82de6b7de54d3bb578be9a00e72e0c21a727db5c46bd3c" gracePeriod=30 Aug 04 15:24:55.292321 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.292288 2559 generic.go:356] "Generic (PLEG): container finished" podID="d0d81a76-367a-4cda-9983-8667444d3c28" containerID="a910d3e9f77a393f9d82de6b7de54d3bb578be9a00e72e0c21a727db5c46bd3c" exitCode=0 Aug 04 15:24:55.292449 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.292353 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" event={"ID":"d0d81a76-367a-4cda-9983-8667444d3c28","Type":"ContainerDied","Data":"a910d3e9f77a393f9d82de6b7de54d3bb578be9a00e72e0c21a727db5c46bd3c"} Aug 04 15:24:55.312221 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.312158 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:24:55.423772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.423743 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7ttd\" (UniqueName: \"kubernetes.io/projected/d0d81a76-367a-4cda-9983-8667444d3c28-kube-api-access-t7ttd\") pod \"d0d81a76-367a-4cda-9983-8667444d3c28\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " Aug 04 15:24:55.423966 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.423785 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert\") pod \"d0d81a76-367a-4cda-9983-8667444d3c28\" (UID: \"d0d81a76-367a-4cda-9983-8667444d3c28\") " Aug 04 15:24:55.425838 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.425808 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d0d81a76-367a-4cda-9983-8667444d3c28" (UID: "d0d81a76-367a-4cda-9983-8667444d3c28"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:24:55.425988 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.425899 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d81a76-367a-4cda-9983-8667444d3c28-kube-api-access-t7ttd" (OuterVolumeSpecName: "kube-api-access-t7ttd") pod "d0d81a76-367a-4cda-9983-8667444d3c28" (UID: "d0d81a76-367a-4cda-9983-8667444d3c28"). InnerVolumeSpecName "kube-api-access-t7ttd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:24:55.525122 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.525092 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ttd\" (UniqueName: \"kubernetes.io/projected/d0d81a76-367a-4cda-9983-8667444d3c28-kube-api-access-t7ttd\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:24:55.525122 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:55.525118 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d0d81a76-367a-4cda-9983-8667444d3c28-webhook-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:24:56.298299 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:56.298267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" event={"ID":"d0d81a76-367a-4cda-9983-8667444d3c28","Type":"ContainerDied","Data":"f4c3f43accac06994b1e82c0276c2d0ee7c7a6b76052e987067fb2797674ef5f"} Aug 04 15:24:56.298299 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:56.298299 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-nwmsb" Aug 04 15:24:56.298747 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:56.298323 2559 scope.go:117] "RemoveContainer" containerID="a910d3e9f77a393f9d82de6b7de54d3bb578be9a00e72e0c21a727db5c46bd3c" Aug 04 15:24:56.326389 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:56.326361 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-nwmsb"] Aug 04 15:24:56.332836 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:56.332817 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-nwmsb"] Aug 04 15:24:57.294881 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:24:57.294841 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0d81a76-367a-4cda-9983-8667444d3c28" path="/var/lib/kubelet/pods/d0d81a76-367a-4cda-9983-8667444d3c28/volumes" Aug 04 15:25:35.120005 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:35.119929 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:25:35.121884 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:35.121861 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:25:35.124730 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:35.124711 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:25:35.126442 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:35.126420 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:25:43.420545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:43.420513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:25:57.155066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.155033 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78bcb9f478-vbjjd"] Aug 04 15:25:57.155529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.155414 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0d81a76-367a-4cda-9983-8667444d3c28" containerName="manager" Aug 04 15:25:57.155529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.155428 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0d81a76-367a-4cda-9983-8667444d3c28" containerName="manager" Aug 04 15:25:57.155529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.155500 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0d81a76-367a-4cda-9983-8667444d3c28" containerName="manager" Aug 04 15:25:57.159693 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.159663 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.163529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.163503 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r9z9v\"" Aug 04 15:25:57.163529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.163518 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 15:25:57.163529 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.163507 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 15:25:57.171535 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.171512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-vbjjd"] Aug 04 15:25:57.258440 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.258404 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tks8\" (UniqueName: \"kubernetes.io/projected/ea990404-87ba-4388-a50b-1cd9e3650cbb-kube-api-access-5tks8\") pod \"maas-controller-78bcb9f478-vbjjd\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.258633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.258475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea990404-87ba-4388-a50b-1cd9e3650cbb-webhook-cert\") pod \"maas-controller-78bcb9f478-vbjjd\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.358925 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.358888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tks8\" (UniqueName: \"kubernetes.io/projected/ea990404-87ba-4388-a50b-1cd9e3650cbb-kube-api-access-5tks8\") pod \"maas-controller-78bcb9f478-vbjjd\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.359092 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.358997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea990404-87ba-4388-a50b-1cd9e3650cbb-webhook-cert\") pod \"maas-controller-78bcb9f478-vbjjd\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.361327 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.361307 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea990404-87ba-4388-a50b-1cd9e3650cbb-webhook-cert\") pod \"maas-controller-78bcb9f478-vbjjd\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.368145 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.368118 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tks8\" (UniqueName: \"kubernetes.io/projected/ea990404-87ba-4388-a50b-1cd9e3650cbb-kube-api-access-5tks8\") pod \"maas-controller-78bcb9f478-vbjjd\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.469841 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.469804 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:57.604734 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:57.604701 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-vbjjd"] Aug 04 15:25:57.605890 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:25:57.605861 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea990404_87ba_4388_a50b_1cd9e3650cbb.slice/crio-e0efdc8becc9bf719be3a8533d0676c41b1892ebb058a3210c3d4675622de8a3 WatchSource:0}: Error finding container e0efdc8becc9bf719be3a8533d0676c41b1892ebb058a3210c3d4675622de8a3: Status 404 returned error can't find the container with id e0efdc8becc9bf719be3a8533d0676c41b1892ebb058a3210c3d4675622de8a3 Aug 04 15:25:58.614247 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:58.614211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" event={"ID":"ea990404-87ba-4388-a50b-1cd9e3650cbb","Type":"ContainerStarted","Data":"0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc"} Aug 04 15:25:58.614643 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:58.614270 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" event={"ID":"ea990404-87ba-4388-a50b-1cd9e3650cbb","Type":"ContainerStarted","Data":"e0efdc8becc9bf719be3a8533d0676c41b1892ebb058a3210c3d4675622de8a3"} Aug 04 15:25:58.614643 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:58.614379 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:25:58.634962 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:25:58.634921 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" podStartSLOduration=1.176418065 podStartE2EDuration="1.634907685s" totalImagesPullingTime="458.48962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:25:58.633811513 +0000 UTC m=+923.987899273" watchObservedRunningTime="2026-08-04 15:25:58.634907685 +0000 UTC m=+923.988995443" Aug 04 15:26:09.621315 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:09.621286 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:26:14.198487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.198454 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:26:14.305139 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.305103 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-vbjjd"] Aug 04 15:26:14.305311 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.305269 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" podUID="ea990404-87ba-4388-a50b-1cd9e3650cbb" containerName="manager" containerID="cri-o://0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc" gracePeriod=30 Aug 04 15:26:14.565179 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.565149 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:26:14.695196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.695160 2559 generic.go:356] "Generic (PLEG): container finished" podID="ea990404-87ba-4388-a50b-1cd9e3650cbb" containerID="0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc" exitCode=0 Aug 04 15:26:14.695391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.695211 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" Aug 04 15:26:14.695391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.695243 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" event={"ID":"ea990404-87ba-4388-a50b-1cd9e3650cbb","Type":"ContainerDied","Data":"0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc"} Aug 04 15:26:14.695391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.695320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-vbjjd" event={"ID":"ea990404-87ba-4388-a50b-1cd9e3650cbb","Type":"ContainerDied","Data":"e0efdc8becc9bf719be3a8533d0676c41b1892ebb058a3210c3d4675622de8a3"} Aug 04 15:26:14.695391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.695343 2559 scope.go:117] "RemoveContainer" containerID="0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc" Aug 04 15:26:14.703440 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.703424 2559 scope.go:117] "RemoveContainer" containerID="0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc" Aug 04 15:26:14.703679 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:26:14.703660 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc\": container with ID starting with 0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc not found: ID does not exist" containerID="0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc" Aug 04 15:26:14.703745 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.703686 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc"} err="failed to get container status \"0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc\": rpc error: code = NotFound desc = could not find container \"0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc\": container with ID starting with 0d5392bf3e1be7c6611e1df7678ac224b8e6434c990f45c43d07efe7155059bc not found: ID does not exist" Aug 04 15:26:14.718004 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.717983 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea990404-87ba-4388-a50b-1cd9e3650cbb-webhook-cert\") pod \"ea990404-87ba-4388-a50b-1cd9e3650cbb\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " Aug 04 15:26:14.718106 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.718061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tks8\" (UniqueName: \"kubernetes.io/projected/ea990404-87ba-4388-a50b-1cd9e3650cbb-kube-api-access-5tks8\") pod \"ea990404-87ba-4388-a50b-1cd9e3650cbb\" (UID: \"ea990404-87ba-4388-a50b-1cd9e3650cbb\") " Aug 04 15:26:14.719866 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.719844 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea990404-87ba-4388-a50b-1cd9e3650cbb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ea990404-87ba-4388-a50b-1cd9e3650cbb" (UID: "ea990404-87ba-4388-a50b-1cd9e3650cbb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:26:14.719946 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.719925 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea990404-87ba-4388-a50b-1cd9e3650cbb-kube-api-access-5tks8" (OuterVolumeSpecName: "kube-api-access-5tks8") pod "ea990404-87ba-4388-a50b-1cd9e3650cbb" (UID: "ea990404-87ba-4388-a50b-1cd9e3650cbb"). InnerVolumeSpecName "kube-api-access-5tks8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:26:14.819555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.819517 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tks8\" (UniqueName: \"kubernetes.io/projected/ea990404-87ba-4388-a50b-1cd9e3650cbb-kube-api-access-5tks8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:26:14.819555 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:14.819551 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea990404-87ba-4388-a50b-1cd9e3650cbb-webhook-cert\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:26:15.017757 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.017726 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-vbjjd"] Aug 04 15:26:15.022005 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.021981 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-vbjjd"] Aug 04 15:26:15.289165 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.289091 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea990404-87ba-4388-a50b-1cd9e3650cbb" path="/var/lib/kubelet/pods/ea990404-87ba-4388-a50b-1cd9e3650cbb/volumes" Aug 04 15:26:15.802835 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.802796 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78bcb9f478-7v569"] Aug 04 15:26:15.803236 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.803223 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea990404-87ba-4388-a50b-1cd9e3650cbb" containerName="manager" Aug 04 15:26:15.803282 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.803238 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea990404-87ba-4388-a50b-1cd9e3650cbb" containerName="manager" Aug 04 15:26:15.803323 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.803317 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea990404-87ba-4388-a50b-1cd9e3650cbb" containerName="manager" Aug 04 15:26:15.807643 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.807628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:15.811836 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.811814 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r9z9v\"" Aug 04 15:26:15.811967 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.811845 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Aug 04 15:26:15.811967 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.811815 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Aug 04 15:26:15.818344 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.818322 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-7v569"] Aug 04 15:26:15.929879 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.929845 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa61e7b9-3ca3-42cf-bd78-59f3d7f17216-webhook-cert\") pod \"maas-controller-78bcb9f478-7v569\" (UID: \"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216\") " pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:15.930060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:15.929884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h45kj\" (UniqueName: \"kubernetes.io/projected/aa61e7b9-3ca3-42cf-bd78-59f3d7f17216-kube-api-access-h45kj\") pod \"maas-controller-78bcb9f478-7v569\" (UID: \"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216\") " pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:16.030850 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.030813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa61e7b9-3ca3-42cf-bd78-59f3d7f17216-webhook-cert\") pod \"maas-controller-78bcb9f478-7v569\" (UID: \"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216\") " pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:16.030850 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.030855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h45kj\" (UniqueName: \"kubernetes.io/projected/aa61e7b9-3ca3-42cf-bd78-59f3d7f17216-kube-api-access-h45kj\") pod \"maas-controller-78bcb9f478-7v569\" (UID: \"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216\") " pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:16.033309 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.033282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa61e7b9-3ca3-42cf-bd78-59f3d7f17216-webhook-cert\") pod \"maas-controller-78bcb9f478-7v569\" (UID: \"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216\") " pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:16.038817 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.038794 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h45kj\" (UniqueName: \"kubernetes.io/projected/aa61e7b9-3ca3-42cf-bd78-59f3d7f17216-kube-api-access-h45kj\") pod \"maas-controller-78bcb9f478-7v569\" (UID: \"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216\") " pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:16.118004 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.117915 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:16.256683 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.256652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78bcb9f478-7v569"] Aug 04 15:26:16.258212 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:26:16.258184 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa61e7b9_3ca3_42cf_bd78_59f3d7f17216.slice/crio-735c2d05075569be31813d52fded9b4fbd3a4e8167e0d493d836e3a80b78ef9b WatchSource:0}: Error finding container 735c2d05075569be31813d52fded9b4fbd3a4e8167e0d493d836e3a80b78ef9b: Status 404 returned error can't find the container with id 735c2d05075569be31813d52fded9b4fbd3a4e8167e0d493d836e3a80b78ef9b Aug 04 15:26:16.264676 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.264642 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:26:16.708059 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:16.708029 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-7v569" event={"ID":"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216","Type":"ContainerStarted","Data":"735c2d05075569be31813d52fded9b4fbd3a4e8167e0d493d836e3a80b78ef9b"} Aug 04 15:26:17.714674 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:17.714556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78bcb9f478-7v569" event={"ID":"aa61e7b9-3ca3-42cf-bd78-59f3d7f17216","Type":"ContainerStarted","Data":"7b415fa83b8adb44fd6e8f715b8ee9dbc61aef14be60c0a780fd0cbf5e565666"} Aug 04 15:26:17.715105 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:17.714685 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:26:17.733756 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:17.733711 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78bcb9f478-7v569" podStartSLOduration=2.283967698 podStartE2EDuration="2.733695348s" totalImagesPullingTime="449.72765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:26:17.731420082 +0000 UTC m=+943.085507840" watchObservedRunningTime="2026-08-04 15:26:17.733695348 +0000 UTC m=+943.087783108" Aug 04 15:26:28.721749 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:26:28.721717 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78bcb9f478-7v569" Aug 04 15:27:15.302984 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:15.302908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:15.391601 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:15.391554 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:15.635165 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:15.635056 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:21.665860 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:21.665823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:21.775141 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:21.775108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:21.869600 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:21.869531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:22.032758 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:22.032723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:27.764365 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:27.764326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:27.864985 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:27.864953 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:27:53.654487 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:27:53.654453 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:04.861467 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:04.861432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:04.937107 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:04.937065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:05.046453 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:05.046422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:06.013798 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:06.013766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:06.075875 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:06.075843 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:15.473102 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:15.473061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:15.571276 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:15.571240 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:15.671216 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:15.671177 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:28:30.283288 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:28:30.283201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:29:54.492013 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:29:54.491979 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:29:54.592036 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:29:54.591999 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:30:35.150820 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:30:35.150789 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:30:35.157382 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:30:35.157348 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:30:35.158781 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:30:35.158759 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:30:35.162491 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:30:35.162473 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:30:43.287618 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:30:43.287566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:32:40.861370 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:40.861337 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n"] Aug 04 15:32:40.861984 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:40.861525 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" podUID="9e73eaf0-0e1f-4618-9a2e-bb199588548d" containerName="manager" containerID="cri-o://a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180" gracePeriod=10 Aug 04 15:32:41.112382 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.112312 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:32:41.226035 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.226002 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w8qg\" (UniqueName: \"kubernetes.io/projected/9e73eaf0-0e1f-4618-9a2e-bb199588548d-kube-api-access-4w8qg\") pod \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " Aug 04 15:32:41.226201 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.226085 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e73eaf0-0e1f-4618-9a2e-bb199588548d-extensions-socket-volume\") pod \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\" (UID: \"9e73eaf0-0e1f-4618-9a2e-bb199588548d\") " Aug 04 15:32:41.226431 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.226406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e73eaf0-0e1f-4618-9a2e-bb199588548d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9e73eaf0-0e1f-4618-9a2e-bb199588548d" (UID: "9e73eaf0-0e1f-4618-9a2e-bb199588548d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 15:32:41.228008 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.227985 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e73eaf0-0e1f-4618-9a2e-bb199588548d-kube-api-access-4w8qg" (OuterVolumeSpecName: "kube-api-access-4w8qg") pod "9e73eaf0-0e1f-4618-9a2e-bb199588548d" (UID: "9e73eaf0-0e1f-4618-9a2e-bb199588548d"). InnerVolumeSpecName "kube-api-access-4w8qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:32:41.327037 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.327007 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8qg\" (UniqueName: \"kubernetes.io/projected/9e73eaf0-0e1f-4618-9a2e-bb199588548d-kube-api-access-4w8qg\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:32:41.327037 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.327033 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e73eaf0-0e1f-4618-9a2e-bb199588548d-extensions-socket-volume\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:32:41.706408 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.706370 2559 generic.go:356] "Generic (PLEG): container finished" podID="9e73eaf0-0e1f-4618-9a2e-bb199588548d" containerID="a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180" exitCode=0 Aug 04 15:32:41.706571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.706419 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" Aug 04 15:32:41.706571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.706452 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" event={"ID":"9e73eaf0-0e1f-4618-9a2e-bb199588548d","Type":"ContainerDied","Data":"a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180"} Aug 04 15:32:41.706571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.706516 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n" event={"ID":"9e73eaf0-0e1f-4618-9a2e-bb199588548d","Type":"ContainerDied","Data":"498e68d00e5029a80967a4d5300e4c04b7e87e833b5dbf3461c8d850f97f167f"} Aug 04 15:32:41.706571 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.706539 2559 scope.go:117] "RemoveContainer" containerID="a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180" Aug 04 15:32:41.716142 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.716126 2559 scope.go:117] "RemoveContainer" containerID="a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180" Aug 04 15:32:41.716391 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:32:41.716370 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180\": container with ID starting with a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180 not found: ID does not exist" containerID="a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180" Aug 04 15:32:41.716438 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.716402 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180"} err="failed to get container status \"a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180\": rpc error: code = NotFound desc = could not find container \"a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180\": container with ID starting with a7a41f1e6482dbdcb4803f2e556a349e75f846d00b0bff8057b81798f40eb180 not found: ID does not exist" Aug 04 15:32:41.727957 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.727934 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n"] Aug 04 15:32:41.732257 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:41.732238 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-s6s2n"] Aug 04 15:32:43.288483 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:32:43.288450 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e73eaf0-0e1f-4618-9a2e-bb199588548d" path="/var/lib/kubelet/pods/9e73eaf0-0e1f-4618-9a2e-bb199588548d/volumes" Aug 04 15:33:41.577715 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.577682 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6"] Aug 04 15:33:41.578163 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.578067 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e73eaf0-0e1f-4618-9a2e-bb199588548d" containerName="manager" Aug 04 15:33:41.578163 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.578079 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e73eaf0-0e1f-4618-9a2e-bb199588548d" containerName="manager" Aug 04 15:33:41.578163 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.578156 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e73eaf0-0e1f-4618-9a2e-bb199588548d" containerName="manager" Aug 04 15:33:41.581368 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.581349 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:41.584080 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.584057 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-85tjv\"" Aug 04 15:33:41.593739 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.593716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6"] Aug 04 15:33:41.652234 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.652196 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nntx\" (UniqueName: \"kubernetes.io/projected/6367d03a-4f1b-4ebf-b3b7-0cae36607628-kube-api-access-6nntx\") pod \"kuadrant-operator-controller-manager-68cf589589-ftlf6\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:41.652405 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.652331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6367d03a-4f1b-4ebf-b3b7-0cae36607628-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-ftlf6\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:41.752805 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.752769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6367d03a-4f1b-4ebf-b3b7-0cae36607628-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-ftlf6\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:41.752970 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.752824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nntx\" (UniqueName: \"kubernetes.io/projected/6367d03a-4f1b-4ebf-b3b7-0cae36607628-kube-api-access-6nntx\") pod \"kuadrant-operator-controller-manager-68cf589589-ftlf6\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:41.753162 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.753141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6367d03a-4f1b-4ebf-b3b7-0cae36607628-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-ftlf6\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:41.775921 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.775884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nntx\" (UniqueName: \"kubernetes.io/projected/6367d03a-4f1b-4ebf-b3b7-0cae36607628-kube-api-access-6nntx\") pod \"kuadrant-operator-controller-manager-68cf589589-ftlf6\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:41.891346 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:41.891259 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:42.036834 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:42.036789 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6"] Aug 04 15:33:42.038044 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:33:42.038013 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6367d03a_4f1b_4ebf_b3b7_0cae36607628.slice/crio-1d8a17642aa546ad449325509a4c804a3b5a28fa7a030211100a75d201b3b9f2 WatchSource:0}: Error finding container 1d8a17642aa546ad449325509a4c804a3b5a28fa7a030211100a75d201b3b9f2: Status 404 returned error can't find the container with id 1d8a17642aa546ad449325509a4c804a3b5a28fa7a030211100a75d201b3b9f2 Aug 04 15:33:43.016252 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:43.016205 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" event={"ID":"6367d03a-4f1b-4ebf-b3b7-0cae36607628","Type":"ContainerStarted","Data":"9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb"} Aug 04 15:33:43.016703 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:43.016268 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" event={"ID":"6367d03a-4f1b-4ebf-b3b7-0cae36607628","Type":"ContainerStarted","Data":"1d8a17642aa546ad449325509a4c804a3b5a28fa7a030211100a75d201b3b9f2"} Aug 04 15:33:43.016703 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:43.016321 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:33:43.049841 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:43.049788 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" podStartSLOduration=2.049773883 podStartE2EDuration="2.049773883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:33:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:33:43.046570977 +0000 UTC m=+1388.400658737" watchObservedRunningTime="2026-08-04 15:33:43.049773883 +0000 UTC m=+1388.403861641" Aug 04 15:33:54.023813 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:33:54.023783 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:34:40.074846 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:34:40.074765 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:35:35.184026 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:35:35.183998 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:35:35.191365 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:35:35.191342 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:35:35.193213 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:35:35.193191 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:35:35.197358 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:35:35.197340 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:36:50.405472 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:36:50.405433 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:37:17.468422 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:37:17.468386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:37:26.939276 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:37:26.939243 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:38:03.222512 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:38:03.222476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:38:05.105334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:38:05.105299 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:39:26.460827 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:39:26.460744 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:39:26.521117 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:39:26.521085 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:40:35.219308 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:40:35.219273 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:40:35.224259 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:40:35.224236 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:40:35.224393 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:40:35.224271 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:40:35.228831 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:40:35.228812 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:41:22.403398 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:41:22.403357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:41:30.440727 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:41:30.440696 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:41:56.612101 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:41:56.612060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:41:59.643993 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:41:59.643902 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:41:59.753903 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:41:59.753871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:00.126145 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:00.126110 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:06.151705 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:06.151665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:07.621360 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:07.621321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:07.722638 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:07.722571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:09.653739 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:09.653708 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:09.756779 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:09.756734 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:10.775321 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.775287 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql"] Aug 04 15:42:10.778875 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.778853 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.782439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.782414 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-197040d2-serving-cert\"" Aug 04 15:42:10.793379 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.793355 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql"] Aug 04 15:42:10.817219 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.817168 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-openshift-service-ca\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.817420 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.817246 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.817420 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.817305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkh4h\" (UniqueName: \"kubernetes.io/projected/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-kube-api-access-mkh4h\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.836755 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.836719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:10.917890 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.917856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-openshift-service-ca\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.918083 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.917922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.918083 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.917972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkh4h\" (UniqueName: \"kubernetes.io/projected/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-kube-api-access-mkh4h\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.918230 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:42:10.918088 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-serving-cert: secret "maas-api-e2e-ait-197040d2-serving-cert" not found Aug 04 15:42:10.918230 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:42:10.918152 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls podName:6d9530ab-f9bc-47a0-bb94-1a1bca49e967 nodeName:}" failed. No retries permitted until 2026-08-04 15:42:11.418136118 +0000 UTC m=+1896.772223855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls") pod "maas-api-e2e-ait-197040d2-574947bb48-9h2ql" (UID: "6d9530ab-f9bc-47a0-bb94-1a1bca49e967") : secret "maas-api-e2e-ait-197040d2-serving-cert" not found Aug 04 15:42:10.918626 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.918568 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-openshift-service-ca\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:10.933463 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:10.933435 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkh4h\" (UniqueName: \"kubernetes.io/projected/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-kube-api-access-mkh4h\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:11.422522 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.422485 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:11.424906 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.424886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls\") pod \"maas-api-e2e-ait-197040d2-574947bb48-9h2ql\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:11.458044 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.458008 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:11.689926 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.689840 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:11.724897 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.724803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:11.852333 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.852293 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql"] Aug 04 15:42:11.858884 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:42:11.858854 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9530ab_f9bc_47a0_bb94_1a1bca49e967.slice/crio-d5bbb44c842d433549690b6021c22ce2d1d4c977c4d3c79f55b1c9a58e636c54 WatchSource:0}: Error finding container d5bbb44c842d433549690b6021c22ce2d1d4c977c4d3c79f55b1c9a58e636c54: Status 404 returned error can't find the container with id d5bbb44c842d433549690b6021c22ce2d1d4c977c4d3c79f55b1c9a58e636c54 Aug 04 15:42:11.860411 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.860390 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:42:11.900889 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:11.900855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:12.622161 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:12.622061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" event={"ID":"6d9530ab-f9bc-47a0-bb94-1a1bca49e967","Type":"ContainerStarted","Data":"39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6"} Aug 04 15:42:12.622161 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:12.622112 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" event={"ID":"6d9530ab-f9bc-47a0-bb94-1a1bca49e967","Type":"ContainerStarted","Data":"d5bbb44c842d433549690b6021c22ce2d1d4c977c4d3c79f55b1c9a58e636c54"} Aug 04 15:42:12.622404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:12.622178 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:12.651928 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:12.651870 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" podStartSLOduration=2.187596546 podStartE2EDuration="2.651853736s" totalImagesPullingTime="464.25719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:42:12.649240507 +0000 UTC m=+1898.003328268" watchObservedRunningTime="2026-08-04 15:42:12.651853736 +0000 UTC m=+1898.005941495" Aug 04 15:42:18.635547 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:18.635516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:51.315048 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.315013 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql"] Aug 04 15:42:51.315681 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.315288 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" podUID="6d9530ab-f9bc-47a0-bb94-1a1bca49e967" containerName="maas-api" containerID="cri-o://39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6" gracePeriod=30 Aug 04 15:42:51.348703 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.348676 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:51.455831 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.455793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:51.540007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.539974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:51.572926 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.572860 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:51.610609 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.610561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:51.689815 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.689785 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkh4h\" (UniqueName: \"kubernetes.io/projected/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-kube-api-access-mkh4h\") pod \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " Aug 04 15:42:51.690013 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.689896 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls\") pod \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " Aug 04 15:42:51.690013 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.689974 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-openshift-service-ca\") pod \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\" (UID: \"6d9530ab-f9bc-47a0-bb94-1a1bca49e967\") " Aug 04 15:42:51.690339 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.690311 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d9530ab-f9bc-47a0-bb94-1a1bca49e967" (UID: "6d9530ab-f9bc-47a0-bb94-1a1bca49e967"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:42:51.692213 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.692191 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d9530ab-f9bc-47a0-bb94-1a1bca49e967" (UID: "6d9530ab-f9bc-47a0-bb94-1a1bca49e967"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:42:51.693476 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.693459 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-kube-api-access-mkh4h" (OuterVolumeSpecName: "kube-api-access-mkh4h") pod "6d9530ab-f9bc-47a0-bb94-1a1bca49e967" (UID: "6d9530ab-f9bc-47a0-bb94-1a1bca49e967"). InnerVolumeSpecName "kube-api-access-mkh4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:42:51.730035 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.730000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:51.791055 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.791016 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:42:51.791055 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.791046 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkh4h\" (UniqueName: \"kubernetes.io/projected/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-kube-api-access-mkh4h\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:42:51.791055 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.791057 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d9530ab-f9bc-47a0-bb94-1a1bca49e967-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:42:51.820566 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.820526 2559 generic.go:356] "Generic (PLEG): container finished" podID="6d9530ab-f9bc-47a0-bb94-1a1bca49e967" containerID="39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6" exitCode=0 Aug 04 15:42:51.820794 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.820572 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" Aug 04 15:42:51.820794 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.820613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" event={"ID":"6d9530ab-f9bc-47a0-bb94-1a1bca49e967","Type":"ContainerDied","Data":"39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6"} Aug 04 15:42:51.820794 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.820675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql" event={"ID":"6d9530ab-f9bc-47a0-bb94-1a1bca49e967","Type":"ContainerDied","Data":"d5bbb44c842d433549690b6021c22ce2d1d4c977c4d3c79f55b1c9a58e636c54"} Aug 04 15:42:51.820794 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.820728 2559 scope.go:117] "RemoveContainer" containerID="39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6" Aug 04 15:42:51.829952 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.829936 2559 scope.go:117] "RemoveContainer" containerID="39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6" Aug 04 15:42:51.830215 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:42:51.830198 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6\": container with ID starting with 39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6 not found: ID does not exist" containerID="39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6" Aug 04 15:42:51.830283 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.830222 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6"} err="failed to get container status \"39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6\": rpc error: code = NotFound desc = could not find container \"39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6\": container with ID starting with 39cf0ccb0e33804dad1d5f194d7f5529cc4beba26fe4985f5cb9ce498c7626f6 not found: ID does not exist" Aug 04 15:42:51.844762 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.844732 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql"] Aug 04 15:42:51.848841 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.848811 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-197040d2-574947bb48-9h2ql"] Aug 04 15:42:51.995067 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:51.995033 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:52.245689 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:52.245651 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:52.340753 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:52.340720 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:52.440092 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:52.440058 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:52.795065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:52.795030 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:53.289939 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:53.289901 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d9530ab-f9bc-47a0-bb94-1a1bca49e967" path="/var/lib/kubelet/pods/6d9530ab-f9bc-47a0-bb94-1a1bca49e967/volumes" Aug 04 15:42:53.472466 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:53.472432 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:54.986363 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:54.986326 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:55.234521 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.234485 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf"] Aug 04 15:42:55.234938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.234924 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d9530ab-f9bc-47a0-bb94-1a1bca49e967" containerName="maas-api" Aug 04 15:42:55.234983 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.234940 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d9530ab-f9bc-47a0-bb94-1a1bca49e967" containerName="maas-api" Aug 04 15:42:55.235018 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.235013 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d9530ab-f9bc-47a0-bb94-1a1bca49e967" containerName="maas-api" Aug 04 15:42:55.239702 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.239638 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.245745 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.245716 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-25b12afe-serving-cert\"" Aug 04 15:42:55.255723 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.255696 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf"] Aug 04 15:42:55.282267 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.282226 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:55.328622 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.328568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752gn\" (UniqueName: \"kubernetes.io/projected/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-kube-api-access-752gn\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.328622 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.328623 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-maas-api-tls\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.328833 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.328713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.429822 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.429789 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752gn\" (UniqueName: \"kubernetes.io/projected/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-kube-api-access-752gn\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.429984 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.429828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-maas-api-tls\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.429984 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.429868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.430473 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.430453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.432257 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.432229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-maas-api-tls\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.439748 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.439718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-752gn\" (UniqueName: \"kubernetes.io/projected/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-kube-api-access-752gn\") pod \"maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.450885 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.450859 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:55.549985 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.549899 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:55.698484 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.698453 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf"] Aug 04 15:42:55.700397 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:42:55.700374 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6776f0_b714_4612_8dbc_cdcd8708e5d8.slice/crio-5a07da6f6829c8473c04f087ff4d110b18bd7f3434f7a5988a6af949c3b844e8 WatchSource:0}: Error finding container 5a07da6f6829c8473c04f087ff4d110b18bd7f3434f7a5988a6af949c3b844e8: Status 404 returned error can't find the container with id 5a07da6f6829c8473c04f087ff4d110b18bd7f3434f7a5988a6af949c3b844e8 Aug 04 15:42:55.807894 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.807803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:55.842823 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.842788 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" event={"ID":"1b6776f0-b714-4612-8dbc-cdcd8708e5d8","Type":"ContainerStarted","Data":"5a07da6f6829c8473c04f087ff4d110b18bd7f3434f7a5988a6af949c3b844e8"} Aug 04 15:42:55.898182 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:55.898149 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:42:57.862333 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:57.862293 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" event={"ID":"1b6776f0-b714-4612-8dbc-cdcd8708e5d8","Type":"ContainerStarted","Data":"4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e"} Aug 04 15:42:57.862767 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:57.862396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:42:57.879938 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:42:57.879879 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" podStartSLOduration=0.79435097 podStartE2EDuration="2.87986464s" totalImagesPullingTime="2.08551367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:42:57.879252459 +0000 UTC m=+1943.233340219" watchObservedRunningTime="2026-08-04 15:42:57.87986464 +0000 UTC m=+1943.233952399" Aug 04 15:43:03.874288 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:03.874255 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:43:26.520065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.520033 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf"] Aug 04 15:43:26.520623 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.520238 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" podUID="1b6776f0-b714-4612-8dbc-cdcd8708e5d8" containerName="maas-api" containerID="cri-o://4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e" gracePeriod=30 Aug 04 15:43:26.541949 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.541921 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:26.624460 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.624420 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:26.705178 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.705150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:26.783472 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.783437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:26.787767 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.787744 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:43:26.912842 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.912814 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-openshift-service-ca\") pod \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " Aug 04 15:43:26.913027 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.912875 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-maas-api-tls\") pod \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " Aug 04 15:43:26.913027 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.912963 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-752gn\" (UniqueName: \"kubernetes.io/projected/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-kube-api-access-752gn\") pod \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\" (UID: \"1b6776f0-b714-4612-8dbc-cdcd8708e5d8\") " Aug 04 15:43:26.913741 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.913653 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b6776f0-b714-4612-8dbc-cdcd8708e5d8" (UID: "1b6776f0-b714-4612-8dbc-cdcd8708e5d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:43:26.916729 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.916707 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-kube-api-access-752gn" (OuterVolumeSpecName: "kube-api-access-752gn") pod "1b6776f0-b714-4612-8dbc-cdcd8708e5d8" (UID: "1b6776f0-b714-4612-8dbc-cdcd8708e5d8"). InnerVolumeSpecName "kube-api-access-752gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:43:26.918065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.918041 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b6776f0-b714-4612-8dbc-cdcd8708e5d8" (UID: "1b6776f0-b714-4612-8dbc-cdcd8708e5d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:43:26.995860 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:26.995827 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:27.011804 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.011772 2559 generic.go:356] "Generic (PLEG): container finished" podID="1b6776f0-b714-4612-8dbc-cdcd8708e5d8" containerID="4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e" exitCode=0 Aug 04 15:43:27.011981 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.011816 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" Aug 04 15:43:27.011981 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.011856 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" event={"ID":"1b6776f0-b714-4612-8dbc-cdcd8708e5d8","Type":"ContainerDied","Data":"4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e"} Aug 04 15:43:27.011981 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.011911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf" event={"ID":"1b6776f0-b714-4612-8dbc-cdcd8708e5d8","Type":"ContainerDied","Data":"5a07da6f6829c8473c04f087ff4d110b18bd7f3434f7a5988a6af949c3b844e8"} Aug 04 15:43:27.011981 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.011941 2559 scope.go:117] "RemoveContainer" containerID="4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e" Aug 04 15:43:27.013652 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.013633 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:43:27.013731 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.013655 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:43:27.013731 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.013665 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752gn\" (UniqueName: \"kubernetes.io/projected/1b6776f0-b714-4612-8dbc-cdcd8708e5d8-kube-api-access-752gn\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:43:27.021621 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.021554 2559 scope.go:117] "RemoveContainer" containerID="4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e" Aug 04 15:43:27.021833 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:43:27.021815 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e\": container with ID starting with 4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e not found: ID does not exist" containerID="4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e" Aug 04 15:43:27.021876 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.021841 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e"} err="failed to get container status \"4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e\": rpc error: code = NotFound desc = could not find container \"4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e\": container with ID starting with 4b85d0fe0b5adb2ba5012a33475fd7dd5fc54926c331881d646a0c337620031e not found: ID does not exist" Aug 04 15:43:27.052876 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.052844 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf"] Aug 04 15:43:27.059851 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.059826 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-25b12afe-6bcf4bcb86-74tkf"] Aug 04 15:43:27.192388 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.192352 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:27.288457 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.288378 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b6776f0-b714-4612-8dbc-cdcd8708e5d8" path="/var/lib/kubelet/pods/1b6776f0-b714-4612-8dbc-cdcd8708e5d8/volumes" Aug 04 15:43:27.410718 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.410640 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:27.792740 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:27.792702 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:28.030825 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:28.030794 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:30.076148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:30.076115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:30.180570 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:30.180530 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:31.175894 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.175859 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x"] Aug 04 15:43:31.176352 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.176292 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b6776f0-b714-4612-8dbc-cdcd8708e5d8" containerName="maas-api" Aug 04 15:43:31.176352 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.176304 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b6776f0-b714-4612-8dbc-cdcd8708e5d8" containerName="maas-api" Aug 04 15:43:31.176470 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.176373 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b6776f0-b714-4612-8dbc-cdcd8708e5d8" containerName="maas-api" Aug 04 15:43:31.180889 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.180869 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.183915 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.183893 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-018493d3-serving-cert\"" Aug 04 15:43:31.196077 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.196054 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x"] Aug 04 15:43:31.211964 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.211934 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:31.250296 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.250259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec176e5b-b16b-41ec-9c18-45e696a5fee4-openshift-service-ca\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.250469 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.250321 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec176e5b-b16b-41ec-9c18-45e696a5fee4-maas-api-tls\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.250469 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.250387 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/ec176e5b-b16b-41ec-9c18-45e696a5fee4-kube-api-access-b6prx\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.351021 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.350988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/ec176e5b-b16b-41ec-9c18-45e696a5fee4-kube-api-access-b6prx\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.351239 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.351051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec176e5b-b16b-41ec-9c18-45e696a5fee4-openshift-service-ca\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.351239 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.351091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec176e5b-b16b-41ec-9c18-45e696a5fee4-maas-api-tls\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.351866 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.351847 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec176e5b-b16b-41ec-9c18-45e696a5fee4-openshift-service-ca\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.353626 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.353605 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec176e5b-b16b-41ec-9c18-45e696a5fee4-maas-api-tls\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.364998 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.364969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/ec176e5b-b16b-41ec-9c18-45e696a5fee4-kube-api-access-b6prx\") pod \"maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.369062 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.369027 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:31.492148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.492112 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:31.644532 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.644497 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x"] Aug 04 15:43:31.648105 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:43:31.648080 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec176e5b_b16b_41ec_9c18_45e696a5fee4.slice/crio-b1ca4f40a8308fbb8e66ccfa2355bdf5e79c087f2c48ae7cb69905934b630344 WatchSource:0}: Error finding container b1ca4f40a8308fbb8e66ccfa2355bdf5e79c087f2c48ae7cb69905934b630344: Status 404 returned error can't find the container with id b1ca4f40a8308fbb8e66ccfa2355bdf5e79c087f2c48ae7cb69905934b630344 Aug 04 15:43:31.722471 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.722442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:31.822886 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:31.822798 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:43:32.039340 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:32.039306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" event={"ID":"ec176e5b-b16b-41ec-9c18-45e696a5fee4","Type":"ContainerStarted","Data":"b1ca4f40a8308fbb8e66ccfa2355bdf5e79c087f2c48ae7cb69905934b630344"} Aug 04 15:43:33.046360 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:33.046326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" event={"ID":"ec176e5b-b16b-41ec-9c18-45e696a5fee4","Type":"ContainerStarted","Data":"d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4"} Aug 04 15:43:33.046821 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:33.046461 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:43:33.063769 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:33.063719 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" podStartSLOduration=1.610861895 podStartE2EDuration="2.063700638s" totalImagesPullingTime="452.838743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:43:33.063146405 +0000 UTC m=+1978.417234170" watchObservedRunningTime="2026-08-04 15:43:33.063700638 +0000 UTC m=+1978.417788398" Aug 04 15:43:39.057317 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:43:39.057282 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:44:01.761757 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:01.761723 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x"] Aug 04 15:44:01.762568 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:01.762508 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" podUID="ec176e5b-b16b-41ec-9c18-45e696a5fee4" containerName="maas-api" containerID="cri-o://d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4" gracePeriod=30 Aug 04 15:44:01.808665 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:01.808631 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:01.910094 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:01.910062 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:02.008600 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.008552 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:44:02.012103 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.012034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:02.091225 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.091193 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/ec176e5b-b16b-41ec-9c18-45e696a5fee4-kube-api-access-b6prx\") pod \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " Aug 04 15:44:02.091412 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.091251 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec176e5b-b16b-41ec-9c18-45e696a5fee4-maas-api-tls\") pod \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " Aug 04 15:44:02.091412 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.091290 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec176e5b-b16b-41ec-9c18-45e696a5fee4-openshift-service-ca\") pod \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\" (UID: \"ec176e5b-b16b-41ec-9c18-45e696a5fee4\") " Aug 04 15:44:02.091932 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.091898 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec176e5b-b16b-41ec-9c18-45e696a5fee4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec176e5b-b16b-41ec-9c18-45e696a5fee4" (UID: "ec176e5b-b16b-41ec-9c18-45e696a5fee4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:44:02.092073 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.091950 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:02.093923 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.093902 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec176e5b-b16b-41ec-9c18-45e696a5fee4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec176e5b-b16b-41ec-9c18-45e696a5fee4" (UID: "ec176e5b-b16b-41ec-9c18-45e696a5fee4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:44:02.093923 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.093913 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec176e5b-b16b-41ec-9c18-45e696a5fee4-kube-api-access-b6prx" (OuterVolumeSpecName: "kube-api-access-b6prx") pod "ec176e5b-b16b-41ec-9c18-45e696a5fee4" (UID: "ec176e5b-b16b-41ec-9c18-45e696a5fee4"). InnerVolumeSpecName "kube-api-access-b6prx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:44:02.188915 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.188859 2559 generic.go:356] "Generic (PLEG): container finished" podID="ec176e5b-b16b-41ec-9c18-45e696a5fee4" containerID="d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4" exitCode=0 Aug 04 15:44:02.188915 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.188913 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" Aug 04 15:44:02.189140 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.188929 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" event={"ID":"ec176e5b-b16b-41ec-9c18-45e696a5fee4","Type":"ContainerDied","Data":"d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4"} Aug 04 15:44:02.189140 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.188988 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x" event={"ID":"ec176e5b-b16b-41ec-9c18-45e696a5fee4","Type":"ContainerDied","Data":"b1ca4f40a8308fbb8e66ccfa2355bdf5e79c087f2c48ae7cb69905934b630344"} Aug 04 15:44:02.189140 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.189016 2559 scope.go:117] "RemoveContainer" containerID="d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4" Aug 04 15:44:02.192859 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.192831 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6prx\" (UniqueName: \"kubernetes.io/projected/ec176e5b-b16b-41ec-9c18-45e696a5fee4-kube-api-access-b6prx\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:44:02.192975 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.192862 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec176e5b-b16b-41ec-9c18-45e696a5fee4-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:44:02.192975 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.192876 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec176e5b-b16b-41ec-9c18-45e696a5fee4-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:44:02.199037 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.199019 2559 scope.go:117] "RemoveContainer" containerID="d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4" Aug 04 15:44:02.199316 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:44:02.199297 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4\": container with ID starting with d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4 not found: ID does not exist" containerID="d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4" Aug 04 15:44:02.199364 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.199326 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4"} err="failed to get container status \"d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4\": rpc error: code = NotFound desc = could not find container \"d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4\": container with ID starting with d9d29057d1422e4a4ba08c405d684f340ab0139032930825fe1227adaa74c6c4 not found: ID does not exist" Aug 04 15:44:02.213518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.213490 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x"] Aug 04 15:44:02.219794 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:02.219768 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-018493d3-794c9c75d5-l9d6x"] Aug 04 15:44:03.203723 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:03.203691 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:03.289302 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:03.289270 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec176e5b-b16b-41ec-9c18-45e696a5fee4" path="/var/lib/kubelet/pods/ec176e5b-b16b-41ec-9c18-45e696a5fee4/volumes" Aug 04 15:44:03.430757 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:03.430709 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:03.646004 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:03.645915 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:03.957235 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:03.957200 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:04.277373 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:04.277281 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:06.303151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:06.303107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:06.396848 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:06.396814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:07.389259 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.389227 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk"] Aug 04 15:44:07.389958 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.389937 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec176e5b-b16b-41ec-9c18-45e696a5fee4" containerName="maas-api" Aug 04 15:44:07.390047 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.389962 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec176e5b-b16b-41ec-9c18-45e696a5fee4" containerName="maas-api" Aug 04 15:44:07.390116 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.390056 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec176e5b-b16b-41ec-9c18-45e696a5fee4" containerName="maas-api" Aug 04 15:44:07.394746 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.394727 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.397133 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.397112 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-cde38272-serving-cert\"" Aug 04 15:44:07.403665 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.403615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk"] Aug 04 15:44:07.420334 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.420302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:07.443872 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.443843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rtdq\" (UniqueName: \"kubernetes.io/projected/57104982-4508-4b63-9b27-2b0b20729d25-kube-api-access-8rtdq\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.444036 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.443925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57104982-4508-4b63-9b27-2b0b20729d25-maas-api-tls\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.444036 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.443945 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57104982-4508-4b63-9b27-2b0b20729d25-openshift-service-ca\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.545018 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.544985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rtdq\" (UniqueName: \"kubernetes.io/projected/57104982-4508-4b63-9b27-2b0b20729d25-kube-api-access-8rtdq\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.545221 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.545070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57104982-4508-4b63-9b27-2b0b20729d25-maas-api-tls\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.545221 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.545092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57104982-4508-4b63-9b27-2b0b20729d25-openshift-service-ca\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.545730 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.545708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57104982-4508-4b63-9b27-2b0b20729d25-openshift-service-ca\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.547606 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.547569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57104982-4508-4b63-9b27-2b0b20729d25-maas-api-tls\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.556456 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.556427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rtdq\" (UniqueName: \"kubernetes.io/projected/57104982-4508-4b63-9b27-2b0b20729d25-kube-api-access-8rtdq\") pod \"maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.706987 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.706955 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:07.851407 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.851382 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk"] Aug 04 15:44:07.853700 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:44:07.853670 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57104982_4508_4b63_9b27_2b0b20729d25.slice/crio-a17b83fabf4dc2a834f488f46086275551097e6987b36d192384d6daf6ecae0c WatchSource:0}: Error finding container a17b83fabf4dc2a834f488f46086275551097e6987b36d192384d6daf6ecae0c: Status 404 returned error can't find the container with id a17b83fabf4dc2a834f488f46086275551097e6987b36d192384d6daf6ecae0c Aug 04 15:44:07.915619 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:07.915569 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:08.013330 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:08.013235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:08.114066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:08.114035 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:08.222430 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:08.222395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" event={"ID":"57104982-4508-4b63-9b27-2b0b20729d25","Type":"ContainerStarted","Data":"a17b83fabf4dc2a834f488f46086275551097e6987b36d192384d6daf6ecae0c"} Aug 04 15:44:09.228956 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:09.228919 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" event={"ID":"57104982-4508-4b63-9b27-2b0b20729d25","Type":"ContainerStarted","Data":"bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237"} Aug 04 15:44:09.229373 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:09.229015 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:09.251424 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:09.251378 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" podStartSLOduration=1.176109508 podStartE2EDuration="2.25136161s" totalImagesPullingTime="1.075252102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:44:09.249317507 +0000 UTC m=+2014.603405265" watchObservedRunningTime="2026-08-04 15:44:09.25136161 +0000 UTC m=+2014.605449368" Aug 04 15:44:15.240732 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:15.240701 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:36.065099 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:36.065063 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:36.622494 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:36.622463 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:36.685154 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:36.685119 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:38.077245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.077212 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk"] Aug 04 15:44:38.077737 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.077458 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" podUID="57104982-4508-4b63-9b27-2b0b20729d25" containerName="maas-api" containerID="cri-o://bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237" gracePeriod=30 Aug 04 15:44:38.104798 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.104766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:38.177305 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.177266 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:38.292109 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.292076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:38.333199 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.333124 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:38.377148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.377116 2559 generic.go:356] "Generic (PLEG): container finished" podID="57104982-4508-4b63-9b27-2b0b20729d25" containerID="bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237" exitCode=0 Aug 04 15:44:38.377341 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.377165 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" Aug 04 15:44:38.377341 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.377195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" event={"ID":"57104982-4508-4b63-9b27-2b0b20729d25","Type":"ContainerDied","Data":"bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237"} Aug 04 15:44:38.377341 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.377246 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk" event={"ID":"57104982-4508-4b63-9b27-2b0b20729d25","Type":"ContainerDied","Data":"a17b83fabf4dc2a834f488f46086275551097e6987b36d192384d6daf6ecae0c"} Aug 04 15:44:38.377341 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.377270 2559 scope.go:117] "RemoveContainer" containerID="bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237" Aug 04 15:44:38.385998 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.385982 2559 scope.go:117] "RemoveContainer" containerID="bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237" Aug 04 15:44:38.386250 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:44:38.386229 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237\": container with ID starting with bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237 not found: ID does not exist" containerID="bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237" Aug 04 15:44:38.386303 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.386267 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237"} err="failed to get container status \"bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237\": rpc error: code = NotFound desc = could not find container \"bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237\": container with ID starting with bfed7860141e42192f97800b5f403d7e839932d51fbacb64510b8475cea99237 not found: ID does not exist" Aug 04 15:44:38.427683 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.427654 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57104982-4508-4b63-9b27-2b0b20729d25-openshift-service-ca\") pod \"57104982-4508-4b63-9b27-2b0b20729d25\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " Aug 04 15:44:38.427865 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.427766 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57104982-4508-4b63-9b27-2b0b20729d25-maas-api-tls\") pod \"57104982-4508-4b63-9b27-2b0b20729d25\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " Aug 04 15:44:38.427865 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.427798 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rtdq\" (UniqueName: \"kubernetes.io/projected/57104982-4508-4b63-9b27-2b0b20729d25-kube-api-access-8rtdq\") pod \"57104982-4508-4b63-9b27-2b0b20729d25\" (UID: \"57104982-4508-4b63-9b27-2b0b20729d25\") " Aug 04 15:44:38.428082 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.428057 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57104982-4508-4b63-9b27-2b0b20729d25-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57104982-4508-4b63-9b27-2b0b20729d25" (UID: "57104982-4508-4b63-9b27-2b0b20729d25"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:44:38.431336 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.431311 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57104982-4508-4b63-9b27-2b0b20729d25-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57104982-4508-4b63-9b27-2b0b20729d25" (UID: "57104982-4508-4b63-9b27-2b0b20729d25"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:44:38.431462 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.431340 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57104982-4508-4b63-9b27-2b0b20729d25-kube-api-access-8rtdq" (OuterVolumeSpecName: "kube-api-access-8rtdq") pod "57104982-4508-4b63-9b27-2b0b20729d25" (UID: "57104982-4508-4b63-9b27-2b0b20729d25"). InnerVolumeSpecName "kube-api-access-8rtdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:44:38.432181 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.432157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:38.529231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.529188 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57104982-4508-4b63-9b27-2b0b20729d25-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:44:38.529231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.529224 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rtdq\" (UniqueName: \"kubernetes.io/projected/57104982-4508-4b63-9b27-2b0b20729d25-kube-api-access-8rtdq\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:44:38.529231 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.529235 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57104982-4508-4b63-9b27-2b0b20729d25-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:44:38.533861 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.533833 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:38.661066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.660967 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:38.700229 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.700197 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk"] Aug 04 15:44:38.706309 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:38.706282 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-cde38272-799b58d8bd-xr7wk"] Aug 04 15:44:39.122809 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:39.122775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:39.223869 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:39.223836 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:39.289087 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:39.289053 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57104982-4508-4b63-9b27-2b0b20729d25" path="/var/lib/kubelet/pods/57104982-4508-4b63-9b27-2b0b20729d25/volumes" Aug 04 15:44:39.326503 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:39.326474 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:50.941940 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:50.941902 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:57.541308 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:57.541221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:57.654558 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:57.654513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:57.745663 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:57.745626 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:57.844880 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:57.844793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:44:58.227425 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:44:58.227389 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:24.179034 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:24.178993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:30.230137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:30.230101 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:30.333860 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:30.333825 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:30.441116 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:30.441079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:35.255562 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:35.255529 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:45:35.259837 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:35.259810 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:45:35.262853 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:35.262830 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:45:35.267093 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:35.267076 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:45:46.335773 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:46.335715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:52.102776 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:52.102732 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:52.207365 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:52.207334 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:45:52.303217 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:45:52.303179 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:02.833750 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:02.833709 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:13.899493 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:13.899459 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:20.262380 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:20.262346 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:20.383978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:20.383933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:20.482127 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:20.482090 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:20.581718 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:20.581603 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:21.210064 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:21.210029 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:21.317765 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:21.317728 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:21.530217 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:21.530129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:22.935769 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:22.935733 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:29.549151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:29.549071 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:31.492117 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:31.492082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:39.352649 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:39.352614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:39.450071 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:39.450040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:40.701103 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:40.701060 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:40.795519 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:40.795469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:41.912676 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.912635 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb"] Aug 04 15:46:41.913165 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.913072 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57104982-4508-4b63-9b27-2b0b20729d25" containerName="maas-api" Aug 04 15:46:41.913165 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.913084 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="57104982-4508-4b63-9b27-2b0b20729d25" containerName="maas-api" Aug 04 15:46:41.913310 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.913185 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="57104982-4508-4b63-9b27-2b0b20729d25" containerName="maas-api" Aug 04 15:46:41.916327 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.916306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:41.918774 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.918750 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ff74c09a-serving-cert\"" Aug 04 15:46:41.929338 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.929311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb"] Aug 04 15:46:41.951273 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:41.951237 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:42.064868 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.064829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnxv\" (UniqueName: \"kubernetes.io/projected/a29b4127-3b31-423d-8ee8-1e5964452482-kube-api-access-whnxv\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.065082 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.064898 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29b4127-3b31-423d-8ee8-1e5964452482-maas-api-tls\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.065082 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.064988 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29b4127-3b31-423d-8ee8-1e5964452482-openshift-service-ca\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.165720 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.165613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29b4127-3b31-423d-8ee8-1e5964452482-openshift-service-ca\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.165720 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.165689 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnxv\" (UniqueName: \"kubernetes.io/projected/a29b4127-3b31-423d-8ee8-1e5964452482-kube-api-access-whnxv\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.165973 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.165746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29b4127-3b31-423d-8ee8-1e5964452482-maas-api-tls\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.166256 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.166233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29b4127-3b31-423d-8ee8-1e5964452482-openshift-service-ca\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.168310 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.168288 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29b4127-3b31-423d-8ee8-1e5964452482-maas-api-tls\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.173913 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.173886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnxv\" (UniqueName: \"kubernetes.io/projected/a29b4127-3b31-423d-8ee8-1e5964452482-kube-api-access-whnxv\") pod \"maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.183948 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.183926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:42.229796 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.229762 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:42.374153 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.374119 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb"] Aug 04 15:46:42.375895 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:46:42.375868 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29b4127_3b31_423d_8ee8_1e5964452482.slice/crio-c35e0a9623d8070e5e41c3c9efd29fc4e34518c765af8989dbb4651286495ac4 WatchSource:0}: Error finding container c35e0a9623d8070e5e41c3c9efd29fc4e34518c765af8989dbb4651286495ac4: Status 404 returned error can't find the container with id c35e0a9623d8070e5e41c3c9efd29fc4e34518c765af8989dbb4651286495ac4 Aug 04 15:46:42.481191 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.481158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:42.556201 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.556163 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:42.639445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:42.639416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:46:43.019923 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:43.019896 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" event={"ID":"a29b4127-3b31-423d-8ee8-1e5964452482","Type":"ContainerStarted","Data":"c35e0a9623d8070e5e41c3c9efd29fc4e34518c765af8989dbb4651286495ac4"} Aug 04 15:46:44.026210 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:44.026171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" event={"ID":"a29b4127-3b31-423d-8ee8-1e5964452482","Type":"ContainerStarted","Data":"f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c"} Aug 04 15:46:44.026618 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:44.026253 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:46:44.042685 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:44.042626 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" podStartSLOduration=2.448079941 podStartE2EDuration="3.042603645s" totalImagesPullingTime="594.523704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:46:44.041662474 +0000 UTC m=+2169.395750233" watchObservedRunningTime="2026-08-04 15:46:44.042603645 +0000 UTC m=+2169.396691406" Aug 04 15:46:50.037458 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:46:50.037429 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:47:05.668556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:05.668514 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:06.388315 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:06.388277 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:06.479653 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:06.479610 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:06.797628 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:06.797594 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:12.903532 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:12.903497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:13.002003 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:13.001956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:17.933240 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:17.933199 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:23.870679 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:23.870640 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb"] Aug 04 15:47:23.873067 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:23.870857 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" podUID="a29b4127-3b31-423d-8ee8-1e5964452482" containerName="maas-api" containerID="cri-o://f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c" gracePeriod=30 Aug 04 15:47:23.899418 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:23.899386 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:23.968332 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:23.968302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:24.031506 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.031469 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:24.119900 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.119862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:24.131324 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.131267 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:47:24.232389 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.232356 2559 generic.go:356] "Generic (PLEG): container finished" podID="a29b4127-3b31-423d-8ee8-1e5964452482" containerID="f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c" exitCode=0 Aug 04 15:47:24.232565 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.232402 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" Aug 04 15:47:24.232565 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.232446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" event={"ID":"a29b4127-3b31-423d-8ee8-1e5964452482","Type":"ContainerDied","Data":"f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c"} Aug 04 15:47:24.232565 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.232503 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb" event={"ID":"a29b4127-3b31-423d-8ee8-1e5964452482","Type":"ContainerDied","Data":"c35e0a9623d8070e5e41c3c9efd29fc4e34518c765af8989dbb4651286495ac4"} Aug 04 15:47:24.232565 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.232527 2559 scope.go:117] "RemoveContainer" containerID="f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c" Aug 04 15:47:24.241904 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.241881 2559 scope.go:117] "RemoveContainer" containerID="f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c" Aug 04 15:47:24.242214 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:47:24.242195 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c\": container with ID starting with f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c not found: ID does not exist" containerID="f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c" Aug 04 15:47:24.242284 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.242222 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c"} err="failed to get container status \"f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c\": rpc error: code = NotFound desc = could not find container \"f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c\": container with ID starting with f47a4ec49eeb1b62ce5317fba669567589a28ac3a9ea13dfac921cb6d007418c not found: ID does not exist" Aug 04 15:47:24.250502 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.250480 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whnxv\" (UniqueName: \"kubernetes.io/projected/a29b4127-3b31-423d-8ee8-1e5964452482-kube-api-access-whnxv\") pod \"a29b4127-3b31-423d-8ee8-1e5964452482\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " Aug 04 15:47:24.250602 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.250535 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29b4127-3b31-423d-8ee8-1e5964452482-maas-api-tls\") pod \"a29b4127-3b31-423d-8ee8-1e5964452482\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " Aug 04 15:47:24.250646 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.250605 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29b4127-3b31-423d-8ee8-1e5964452482-openshift-service-ca\") pod \"a29b4127-3b31-423d-8ee8-1e5964452482\" (UID: \"a29b4127-3b31-423d-8ee8-1e5964452482\") " Aug 04 15:47:24.250977 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.250954 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29b4127-3b31-423d-8ee8-1e5964452482-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a29b4127-3b31-423d-8ee8-1e5964452482" (UID: "a29b4127-3b31-423d-8ee8-1e5964452482"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:47:24.252624 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.252593 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29b4127-3b31-423d-8ee8-1e5964452482-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a29b4127-3b31-423d-8ee8-1e5964452482" (UID: "a29b4127-3b31-423d-8ee8-1e5964452482"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:47:24.252730 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.252637 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29b4127-3b31-423d-8ee8-1e5964452482-kube-api-access-whnxv" (OuterVolumeSpecName: "kube-api-access-whnxv") pod "a29b4127-3b31-423d-8ee8-1e5964452482" (UID: "a29b4127-3b31-423d-8ee8-1e5964452482"). InnerVolumeSpecName "kube-api-access-whnxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:47:24.352312 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.352266 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnxv\" (UniqueName: \"kubernetes.io/projected/a29b4127-3b31-423d-8ee8-1e5964452482-kube-api-access-whnxv\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:47:24.352312 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.352316 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a29b4127-3b31-423d-8ee8-1e5964452482-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:47:24.352527 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.352328 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29b4127-3b31-423d-8ee8-1e5964452482-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:47:24.564758 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.564723 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb"] Aug 04 15:47:24.567081 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.567045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:24.571186 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:24.571165 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ff74c09a-64c4d6f948-qw8lb"] Aug 04 15:47:25.292479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:25.292434 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a29b4127-3b31-423d-8ee8-1e5964452482" path="/var/lib/kubelet/pods/a29b4127-3b31-423d-8ee8-1e5964452482/volumes" Aug 04 15:47:28.521906 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:28.521868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:28.622660 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:28.622625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:30.495214 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:30.495184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:30.600186 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:30.600153 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:30.775405 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:30.775318 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:30.994330 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:30.994295 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:31.148445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:31.148367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:31.248695 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:31.248661 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:31.348426 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:31.348393 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:33.850827 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:33.850792 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:37.465087 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:37.465055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:37.566632 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:37.566566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:37.668077 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:37.668039 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:37.814772 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:37.814681 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:43.850104 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:43.850067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:47:43.930339 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:47:43.930291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:48:12.185256 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.185144 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6"] Aug 04 15:48:12.185847 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.185384 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" podUID="6367d03a-4f1b-4ebf-b3b7-0cae36607628" containerName="manager" containerID="cri-o://9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb" gracePeriod=10 Aug 04 15:48:12.438067 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.438002 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:48:12.480557 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.480513 2559 generic.go:356] "Generic (PLEG): container finished" podID="6367d03a-4f1b-4ebf-b3b7-0cae36607628" containerID="9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb" exitCode=0 Aug 04 15:48:12.480803 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.480682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" event={"ID":"6367d03a-4f1b-4ebf-b3b7-0cae36607628","Type":"ContainerDied","Data":"9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb"} Aug 04 15:48:12.480803 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.480738 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" event={"ID":"6367d03a-4f1b-4ebf-b3b7-0cae36607628","Type":"ContainerDied","Data":"1d8a17642aa546ad449325509a4c804a3b5a28fa7a030211100a75d201b3b9f2"} Aug 04 15:48:12.480803 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.480773 2559 scope.go:117] "RemoveContainer" containerID="9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb" Aug 04 15:48:12.481045 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.480932 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6" Aug 04 15:48:12.493611 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.493567 2559 scope.go:117] "RemoveContainer" containerID="9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb" Aug 04 15:48:12.493906 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:48:12.493887 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb\": container with ID starting with 9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb not found: ID does not exist" containerID="9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb" Aug 04 15:48:12.493979 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.493922 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb"} err="failed to get container status \"9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb\": rpc error: code = NotFound desc = could not find container \"9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb\": container with ID starting with 9df959069aad6be87d043c8d506458b8e815a61f25962d1949022131e593ccdb not found: ID does not exist" Aug 04 15:48:12.495191 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.495173 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nntx\" (UniqueName: \"kubernetes.io/projected/6367d03a-4f1b-4ebf-b3b7-0cae36607628-kube-api-access-6nntx\") pod \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " Aug 04 15:48:12.495245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.495203 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6367d03a-4f1b-4ebf-b3b7-0cae36607628-extensions-socket-volume\") pod \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\" (UID: \"6367d03a-4f1b-4ebf-b3b7-0cae36607628\") " Aug 04 15:48:12.495645 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.495613 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6367d03a-4f1b-4ebf-b3b7-0cae36607628-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6367d03a-4f1b-4ebf-b3b7-0cae36607628" (UID: "6367d03a-4f1b-4ebf-b3b7-0cae36607628"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 15:48:12.497344 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.497318 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6367d03a-4f1b-4ebf-b3b7-0cae36607628-kube-api-access-6nntx" (OuterVolumeSpecName: "kube-api-access-6nntx") pod "6367d03a-4f1b-4ebf-b3b7-0cae36607628" (UID: "6367d03a-4f1b-4ebf-b3b7-0cae36607628"). InnerVolumeSpecName "kube-api-access-6nntx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:48:12.596479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.596437 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nntx\" (UniqueName: \"kubernetes.io/projected/6367d03a-4f1b-4ebf-b3b7-0cae36607628-kube-api-access-6nntx\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:48:12.596479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.596472 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6367d03a-4f1b-4ebf-b3b7-0cae36607628-extensions-socket-volume\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:48:12.808508 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.808471 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6"] Aug 04 15:48:12.817264 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:12.817236 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-ftlf6"] Aug 04 15:48:13.288389 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:48:13.288357 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6367d03a-4f1b-4ebf-b3b7-0cae36607628" path="/var/lib/kubelet/pods/6367d03a-4f1b-4ebf-b3b7-0cae36607628/volumes" Aug 04 15:49:27.809498 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.809418 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42"] Aug 04 15:49:27.809991 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.809810 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29b4127-3b31-423d-8ee8-1e5964452482" containerName="maas-api" Aug 04 15:49:27.809991 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.809823 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29b4127-3b31-423d-8ee8-1e5964452482" containerName="maas-api" Aug 04 15:49:27.809991 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.809842 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6367d03a-4f1b-4ebf-b3b7-0cae36607628" containerName="manager" Aug 04 15:49:27.809991 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.809847 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6367d03a-4f1b-4ebf-b3b7-0cae36607628" containerName="manager" Aug 04 15:49:27.809991 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.809911 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6367d03a-4f1b-4ebf-b3b7-0cae36607628" containerName="manager" Aug 04 15:49:27.809991 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.809918 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29b4127-3b31-423d-8ee8-1e5964452482" containerName="maas-api" Aug 04 15:49:27.812934 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.812919 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:27.816234 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.816212 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-85tjv\"" Aug 04 15:49:27.846097 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.846063 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42"] Aug 04 15:49:27.940833 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.940793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d835dfbd-22a6-41df-872d-5f7652e0a83f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-95p42\" (UID: \"d835dfbd-22a6-41df-872d-5f7652e0a83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:27.941012 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:27.940855 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbs2s\" (UniqueName: \"kubernetes.io/projected/d835dfbd-22a6-41df-872d-5f7652e0a83f-kube-api-access-jbs2s\") pod \"kuadrant-operator-controller-manager-68cf589589-95p42\" (UID: \"d835dfbd-22a6-41df-872d-5f7652e0a83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:28.042011 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.041980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d835dfbd-22a6-41df-872d-5f7652e0a83f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-95p42\" (UID: \"d835dfbd-22a6-41df-872d-5f7652e0a83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:28.042180 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.042037 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbs2s\" (UniqueName: \"kubernetes.io/projected/d835dfbd-22a6-41df-872d-5f7652e0a83f-kube-api-access-jbs2s\") pod \"kuadrant-operator-controller-manager-68cf589589-95p42\" (UID: \"d835dfbd-22a6-41df-872d-5f7652e0a83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:28.042381 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.042359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d835dfbd-22a6-41df-872d-5f7652e0a83f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-95p42\" (UID: \"d835dfbd-22a6-41df-872d-5f7652e0a83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:28.056105 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.056077 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbs2s\" (UniqueName: \"kubernetes.io/projected/d835dfbd-22a6-41df-872d-5f7652e0a83f-kube-api-access-jbs2s\") pod \"kuadrant-operator-controller-manager-68cf589589-95p42\" (UID: \"d835dfbd-22a6-41df-872d-5f7652e0a83f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:28.122212 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.122135 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:28.292902 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:49:28.292863 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd835dfbd_22a6_41df_872d_5f7652e0a83f.slice/crio-2880683735ae8c5bf87d21045be6585879a9094d03a4b4d13a3517bc166f04ce WatchSource:0}: Error finding container 2880683735ae8c5bf87d21045be6585879a9094d03a4b4d13a3517bc166f04ce: Status 404 returned error can't find the container with id 2880683735ae8c5bf87d21045be6585879a9094d03a4b4d13a3517bc166f04ce Aug 04 15:49:28.296499 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.296434 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42"] Aug 04 15:49:28.846668 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.846634 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" event={"ID":"d835dfbd-22a6-41df-872d-5f7652e0a83f","Type":"ContainerStarted","Data":"d38255927af35e23133e0efcd1271c8c0c19a4214a56fb9c9871fe6dfc50b195"} Aug 04 15:49:28.847066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.846692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" event={"ID":"d835dfbd-22a6-41df-872d-5f7652e0a83f","Type":"ContainerStarted","Data":"2880683735ae8c5bf87d21045be6585879a9094d03a4b4d13a3517bc166f04ce"} Aug 04 15:49:28.847066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.846730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:28.876263 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:28.876098 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" podStartSLOduration=1.876080295 podStartE2EDuration="1.876080295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:49:28.874074976 +0000 UTC m=+2334.228162735" watchObservedRunningTime="2026-08-04 15:49:28.876080295 +0000 UTC m=+2334.230168056" Aug 04 15:49:39.853924 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:39.853889 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-95p42" Aug 04 15:49:46.634041 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:46.634003 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:46.972356 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:46.972311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:47.448128 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:47.448094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:49.751478 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:49.751442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:49.849404 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:49.849367 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:50.897217 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.897161 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465"] Aug 04 15:49:50.901950 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.901923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:50.904651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.904626 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e4d56b38-serving-cert\"" Aug 04 15:49:50.920547 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.920514 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465"] Aug 04 15:49:50.940633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.940594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzb5v\" (UniqueName: \"kubernetes.io/projected/63c5fff4-e195-4549-b234-0272974a2b45-kube-api-access-vzb5v\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:50.941100 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.940644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:50.941100 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.940673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c5fff4-e195-4549-b234-0272974a2b45-openshift-service-ca\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:50.948240 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:50.948207 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:51.041520 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.041481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c5fff4-e195-4549-b234-0272974a2b45-openshift-service-ca\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.041728 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.041619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzb5v\" (UniqueName: \"kubernetes.io/projected/63c5fff4-e195-4549-b234-0272974a2b45-kube-api-access-vzb5v\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.041728 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.041651 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.041855 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:49:51.041779 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-serving-cert: secret "maas-api-e2e-mt-e4d56b38-serving-cert" not found Aug 04 15:49:51.041916 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:49:51.041865 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls podName:63c5fff4-e195-4549-b234-0272974a2b45 nodeName:}" failed. No retries permitted until 2026-08-04 15:49:51.541841456 +0000 UTC m=+2356.895929208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls") pod "maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" (UID: "63c5fff4-e195-4549-b234-0272974a2b45") : secret "maas-api-e2e-mt-e4d56b38-serving-cert" not found Aug 04 15:49:51.042414 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.042382 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c5fff4-e195-4549-b234-0272974a2b45-openshift-service-ca\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.055176 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.055120 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzb5v\" (UniqueName: \"kubernetes.io/projected/63c5fff4-e195-4549-b234-0272974a2b45-kube-api-access-vzb5v\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.454467 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.454434 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:51.547216 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.547179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.549788 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.549763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls\") pod \"maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.550002 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.549975 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:49:51.814060 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.813967 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:51.956497 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.956471 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465"] Aug 04 15:49:51.958833 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:49:51.958802 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c5fff4_e195_4549_b234_0272974a2b45.slice/crio-3b90e007c5309a36e11d648605464cbc29bc513e9669c3f3268fe29648af7af0 WatchSource:0}: Error finding container 3b90e007c5309a36e11d648605464cbc29bc513e9669c3f3268fe29648af7af0: Status 404 returned error can't find the container with id 3b90e007c5309a36e11d648605464cbc29bc513e9669c3f3268fe29648af7af0 Aug 04 15:49:51.960428 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.960409 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:49:51.970418 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:51.970385 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" event={"ID":"63c5fff4-e195-4549-b234-0272974a2b45","Type":"ContainerStarted","Data":"3b90e007c5309a36e11d648605464cbc29bc513e9669c3f3268fe29648af7af0"} Aug 04 15:49:52.976764 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:52.976729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" event={"ID":"63c5fff4-e195-4549-b234-0272974a2b45","Type":"ContainerStarted","Data":"997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81"} Aug 04 15:49:52.977192 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:52.976843 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:49:52.995391 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:52.995337 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" podStartSLOduration=2.600438789 podStartE2EDuration="2.995321231s" totalImagesPullingTime="394.882442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:49:52.993865017 +0000 UTC m=+2358.347952778" watchObservedRunningTime="2026-08-04 15:49:52.995321231 +0000 UTC m=+2358.349408990" Aug 04 15:49:58.987279 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:49:58.987247 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:50:14.718593 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:50:14.718541 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:50:25.910143 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:50:25.910109 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:50:35.288111 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:50:35.288082 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:50:35.292495 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:50:35.292474 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:50:35.294146 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:50:35.294124 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:50:35.298914 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:50:35.298897 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:51:09.766989 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:09.766911 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465"] Aug 04 15:51:09.767526 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:09.767170 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" podUID="63c5fff4-e195-4549-b234-0272974a2b45" containerName="maas-api" containerID="cri-o://997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81" gracePeriod=30 Aug 04 15:51:10.025311 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.025246 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:51:10.163085 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.163053 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzb5v\" (UniqueName: \"kubernetes.io/projected/63c5fff4-e195-4549-b234-0272974a2b45-kube-api-access-vzb5v\") pod \"63c5fff4-e195-4549-b234-0272974a2b45\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " Aug 04 15:51:10.163245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.163123 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls\") pod \"63c5fff4-e195-4549-b234-0272974a2b45\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " Aug 04 15:51:10.163245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.163185 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c5fff4-e195-4549-b234-0272974a2b45-openshift-service-ca\") pod \"63c5fff4-e195-4549-b234-0272974a2b45\" (UID: \"63c5fff4-e195-4549-b234-0272974a2b45\") " Aug 04 15:51:10.163611 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.163562 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c5fff4-e195-4549-b234-0272974a2b45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63c5fff4-e195-4549-b234-0272974a2b45" (UID: "63c5fff4-e195-4549-b234-0272974a2b45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:51:10.165229 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.165200 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c5fff4-e195-4549-b234-0272974a2b45-kube-api-access-vzb5v" (OuterVolumeSpecName: "kube-api-access-vzb5v") pod "63c5fff4-e195-4549-b234-0272974a2b45" (UID: "63c5fff4-e195-4549-b234-0272974a2b45"). InnerVolumeSpecName "kube-api-access-vzb5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:51:10.165316 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.165252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63c5fff4-e195-4549-b234-0272974a2b45" (UID: "63c5fff4-e195-4549-b234-0272974a2b45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:51:10.264832 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.264799 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzb5v\" (UniqueName: \"kubernetes.io/projected/63c5fff4-e195-4549-b234-0272974a2b45-kube-api-access-vzb5v\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:51:10.264832 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.264830 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c5fff4-e195-4549-b234-0272974a2b45-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:51:10.264832 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.264838 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c5fff4-e195-4549-b234-0272974a2b45-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:51:10.267826 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.267793 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:51:10.376659 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.376560 2559 generic.go:356] "Generic (PLEG): container finished" podID="63c5fff4-e195-4549-b234-0272974a2b45" containerID="997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81" exitCode=0 Aug 04 15:51:10.376659 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.376605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" event={"ID":"63c5fff4-e195-4549-b234-0272974a2b45","Type":"ContainerDied","Data":"997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81"} Aug 04 15:51:10.376659 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.376622 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" Aug 04 15:51:10.376916 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.376659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465" event={"ID":"63c5fff4-e195-4549-b234-0272974a2b45","Type":"ContainerDied","Data":"3b90e007c5309a36e11d648605464cbc29bc513e9669c3f3268fe29648af7af0"} Aug 04 15:51:10.376916 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.376686 2559 scope.go:117] "RemoveContainer" containerID="997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81" Aug 04 15:51:10.386273 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.386254 2559 scope.go:117] "RemoveContainer" containerID="997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81" Aug 04 15:51:10.386548 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:51:10.386529 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81\": container with ID starting with 997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81 not found: ID does not exist" containerID="997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81" Aug 04 15:51:10.386618 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.386557 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81"} err="failed to get container status \"997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81\": rpc error: code = NotFound desc = could not find container \"997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81\": container with ID starting with 997e47d90a37d582b539c460dc5f830d98bcb6fbce065eabbf5ea3076bffaf81 not found: ID does not exist" Aug 04 15:51:10.399109 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.399086 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465"] Aug 04 15:51:10.405496 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.405475 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4d56b38-97fc86cc5-tn465"] Aug 04 15:51:10.837788 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:10.837757 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:51:11.288331 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:11.288296 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c5fff4-e195-4549-b234-0272974a2b45" path="/var/lib/kubelet/pods/63c5fff4-e195-4549-b234-0272974a2b45/volumes" Aug 04 15:51:13.124316 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:13.124279 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:51:18.202821 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:18.202784 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:51:28.747112 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:28.747081 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:51:49.756676 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:49.756641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:51:54.412349 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:54.412319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:51:55.574721 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:51:55.574674 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:01.196458 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:01.196423 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:24.866252 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:24.866218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:24.968029 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:24.967993 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:25.188633 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:25.188524 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:25.410838 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:25.410803 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:36.709196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:36.709111 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:36.855619 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:36.855566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:36.997423 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:36.997321 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:37.623670 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:37.623622 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:37.713249 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:37.713211 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:37.812714 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:37.812678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:38.126088 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:38.126054 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:38.223739 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:38.223550 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:39.177222 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:39.177189 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:46.564031 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:46.563995 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:46.660788 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:46.660746 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:47.284409 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:47.284370 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:53.414388 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:53.414357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:52:53.583743 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:52:53.583712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:04.052189 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:04.052156 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:04.288012 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:04.287971 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:15.877594 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:15.877532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:25.243905 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:25.243868 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:31.585148 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:31.585108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:31.664927 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:31.664850 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:32.011012 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:32.010977 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:32.269793 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:32.269701 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:32.541077 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:32.540958 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:35.040269 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:35.040230 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:35.118868 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:35.118838 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:36.123754 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.123721 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z"] Aug 04 15:53:36.124309 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.124281 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c5fff4-e195-4549-b234-0272974a2b45" containerName="maas-api" Aug 04 15:53:36.124309 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.124299 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c5fff4-e195-4549-b234-0272974a2b45" containerName="maas-api" Aug 04 15:53:36.124506 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.124431 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c5fff4-e195-4549-b234-0272974a2b45" containerName="maas-api" Aug 04 15:53:36.128308 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.128286 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.134091 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.134070 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-21cd04-serving-cert\"" Aug 04 15:53:36.139757 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.139735 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z"] Aug 04 15:53:36.280547 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.280509 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bc4239-db30-47a4-8268-faa1ece4652f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.280760 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.280557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfhd\" (UniqueName: \"kubernetes.io/projected/93bc4239-db30-47a4-8268-faa1ece4652f-kube-api-access-6pfhd\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.280760 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.280704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.381343 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.381246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.381528 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.381355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bc4239-db30-47a4-8268-faa1ece4652f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.381528 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.381398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfhd\" (UniqueName: \"kubernetes.io/projected/93bc4239-db30-47a4-8268-faa1ece4652f-kube-api-access-6pfhd\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.381528 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:53:36.381404 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-serving-cert: secret "maas-api-e2e-inf-a-21cd04-serving-cert" not found Aug 04 15:53:36.381528 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:53:36.381479 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls podName:93bc4239-db30-47a4-8268-faa1ece4652f nodeName:}" failed. No retries permitted until 2026-08-04 15:53:36.881459852 +0000 UTC m=+2582.235547591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls") pod "maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" (UID: "93bc4239-db30-47a4-8268-faa1ece4652f") : secret "maas-api-e2e-inf-a-21cd04-serving-cert" not found Aug 04 15:53:36.382099 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.382073 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bc4239-db30-47a4-8268-faa1ece4652f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.390193 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.390165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfhd\" (UniqueName: \"kubernetes.io/projected/93bc4239-db30-47a4-8268-faa1ece4652f-kube-api-access-6pfhd\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.885354 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.885316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:36.887946 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:36.887921 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls\") pod \"maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:37.040225 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:37.040187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:37.250296 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:37.250263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:37.350629 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:37.350595 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:53:37.391849 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:37.391824 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z"] Aug 04 15:53:37.394628 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:53:37.394601 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bc4239_db30_47a4_8268_faa1ece4652f.slice/crio-af19cb73afd512b27c676f45dfd452a603ef589e3de2ecb0f81a012c211fa7c2 WatchSource:0}: Error finding container af19cb73afd512b27c676f45dfd452a603ef589e3de2ecb0f81a012c211fa7c2: Status 404 returned error can't find the container with id af19cb73afd512b27c676f45dfd452a603ef589e3de2ecb0f81a012c211fa7c2 Aug 04 15:53:38.126157 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:38.126117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" event={"ID":"93bc4239-db30-47a4-8268-faa1ece4652f","Type":"ContainerStarted","Data":"af19cb73afd512b27c676f45dfd452a603ef589e3de2ecb0f81a012c211fa7c2"} Aug 04 15:53:39.132223 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:39.132189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" event={"ID":"93bc4239-db30-47a4-8268-faa1ece4652f","Type":"ContainerStarted","Data":"f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d"} Aug 04 15:53:39.132651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:39.132299 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:53:39.153023 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:39.152973 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" podStartSLOduration=2.133450286 podStartE2EDuration="3.152957153s" totalImagesPullingTime="1.019506867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:53:39.151444436 +0000 UTC m=+2584.505532197" watchObservedRunningTime="2026-08-04 15:53:39.152957153 +0000 UTC m=+2584.507044911" Aug 04 15:53:45.143052 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:53:45.143023 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:54:00.091398 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:00.091311 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:00.483568 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:00.483534 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:00.834424 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:00.834352 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:02.583516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:02.583485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:04.182784 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:04.182753 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:05.267393 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.267360 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9"] Aug 04 15:54:05.271388 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.271371 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.273833 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.273812 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9d3b05-serving-cert\"" Aug 04 15:54:05.290969 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.290944 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9"] Aug 04 15:54:05.323258 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.323220 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:05.342953 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.342920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqz24\" (UniqueName: \"kubernetes.io/projected/1d2237c4-8b8d-4338-8686-502ebb40c4ff-kube-api-access-cqz24\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.343151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.342959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2237c4-8b8d-4338-8686-502ebb40c4ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.343151 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.343116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.427096 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.427061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:05.443660 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.443628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.443812 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.443687 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqz24\" (UniqueName: \"kubernetes.io/projected/1d2237c4-8b8d-4338-8686-502ebb40c4ff-kube-api-access-cqz24\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.443812 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.443718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2237c4-8b8d-4338-8686-502ebb40c4ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.443812 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:54:05.443786 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-serving-cert: secret "maas-api-e2e-inf-b-9d3b05-serving-cert" not found Aug 04 15:54:05.443925 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:54:05.443870 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls podName:1d2237c4-8b8d-4338-8686-502ebb40c4ff nodeName:}" failed. No retries permitted until 2026-08-04 15:54:05.943849314 +0000 UTC m=+2611.297937051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls") pod "maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" (UID: "1d2237c4-8b8d-4338-8686-502ebb40c4ff") : secret "maas-api-e2e-inf-b-9d3b05-serving-cert" not found Aug 04 15:54:05.444464 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.444444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2237c4-8b8d-4338-8686-502ebb40c4ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.453739 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.453712 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqz24\" (UniqueName: \"kubernetes.io/projected/1d2237c4-8b8d-4338-8686-502ebb40c4ff-kube-api-access-cqz24\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.827150 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.827121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:05.927068 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.927021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:05.947425 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.947383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:05.950065 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:05.950038 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:06.186606 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:06.186488 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:06.328304 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:06.328273 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9"] Aug 04 15:54:06.329674 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:54:06.329650 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2237c4_8b8d_4338_8686_502ebb40c4ff.slice/crio-e12a87663de5f5369c1ae6806db01e3e72a2be60987cf1b468e0d308b099fb02 WatchSource:0}: Error finding container e12a87663de5f5369c1ae6806db01e3e72a2be60987cf1b468e0d308b099fb02: Status 404 returned error can't find the container with id e12a87663de5f5369c1ae6806db01e3e72a2be60987cf1b468e0d308b099fb02 Aug 04 15:54:07.280250 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:07.280206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" event={"ID":"1d2237c4-8b8d-4338-8686-502ebb40c4ff","Type":"ContainerStarted","Data":"e12a87663de5f5369c1ae6806db01e3e72a2be60987cf1b468e0d308b099fb02"} Aug 04 15:54:08.286442 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:08.286405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" event={"ID":"1d2237c4-8b8d-4338-8686-502ebb40c4ff","Type":"ContainerStarted","Data":"90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22"} Aug 04 15:54:08.286850 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:08.286521 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:08.306481 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:08.306416 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" podStartSLOduration=2.30561201 podStartE2EDuration="3.306400767s" totalImagesPullingTime="1.000788757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:54:08.304094683 +0000 UTC m=+2613.658182441" watchObservedRunningTime="2026-08-04 15:54:08.306400767 +0000 UTC m=+2613.660488598" Aug 04 15:54:14.297560 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:14.297530 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:54:29.149211 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:29.149174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:38.621475 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:38.621441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:54:58.521166 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:54:58.521105 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:01.457743 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:01.457700 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:35.325308 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:35.325236 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:55:35.330010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:35.329985 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:55:35.332396 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:35.332369 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 15:55:35.336839 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:35.336821 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 15:55:38.818607 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:38.818552 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:39.723935 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:39.723902 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:50.357556 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.357518 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z"] Aug 04 15:55:50.357947 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.357775 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" podUID="93bc4239-db30-47a4-8268-faa1ece4652f" containerName="maas-api" containerID="cri-o://f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d" gracePeriod=30 Aug 04 15:55:50.371443 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.371415 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:50.473700 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.473665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:50.555780 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.555743 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:50.600952 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.600932 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:55:50.695350 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.695329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls\") pod \"93bc4239-db30-47a4-8268-faa1ece4652f\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " Aug 04 15:55:50.695518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.695366 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bc4239-db30-47a4-8268-faa1ece4652f-openshift-service-ca\") pod \"93bc4239-db30-47a4-8268-faa1ece4652f\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " Aug 04 15:55:50.695518 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.695411 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pfhd\" (UniqueName: \"kubernetes.io/projected/93bc4239-db30-47a4-8268-faa1ece4652f-kube-api-access-6pfhd\") pod \"93bc4239-db30-47a4-8268-faa1ece4652f\" (UID: \"93bc4239-db30-47a4-8268-faa1ece4652f\") " Aug 04 15:55:50.695835 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.695805 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93bc4239-db30-47a4-8268-faa1ece4652f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93bc4239-db30-47a4-8268-faa1ece4652f" (UID: "93bc4239-db30-47a4-8268-faa1ece4652f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:55:50.697479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.697457 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93bc4239-db30-47a4-8268-faa1ece4652f" (UID: "93bc4239-db30-47a4-8268-faa1ece4652f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:55:50.697560 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.697535 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bc4239-db30-47a4-8268-faa1ece4652f-kube-api-access-6pfhd" (OuterVolumeSpecName: "kube-api-access-6pfhd") pod "93bc4239-db30-47a4-8268-faa1ece4652f" (UID: "93bc4239-db30-47a4-8268-faa1ece4652f"). InnerVolumeSpecName "kube-api-access-6pfhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:55:50.795924 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.795894 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93bc4239-db30-47a4-8268-faa1ece4652f-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:55:50.795924 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.795918 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93bc4239-db30-47a4-8268-faa1ece4652f-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:55:50.795924 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.795928 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfhd\" (UniqueName: \"kubernetes.io/projected/93bc4239-db30-47a4-8268-faa1ece4652f-kube-api-access-6pfhd\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:55:50.805317 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.805287 2559 generic.go:356] "Generic (PLEG): container finished" podID="93bc4239-db30-47a4-8268-faa1ece4652f" containerID="f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d" exitCode=0 Aug 04 15:55:50.805447 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.805336 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" Aug 04 15:55:50.805447 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.805374 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" event={"ID":"93bc4239-db30-47a4-8268-faa1ece4652f","Type":"ContainerDied","Data":"f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d"} Aug 04 15:55:50.805447 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.805445 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z" event={"ID":"93bc4239-db30-47a4-8268-faa1ece4652f","Type":"ContainerDied","Data":"af19cb73afd512b27c676f45dfd452a603ef589e3de2ecb0f81a012c211fa7c2"} Aug 04 15:55:50.805613 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.805468 2559 scope.go:117] "RemoveContainer" containerID="f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d" Aug 04 15:55:50.815261 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.815246 2559 scope.go:117] "RemoveContainer" containerID="f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d" Aug 04 15:55:50.815489 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:55:50.815471 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d\": container with ID starting with f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d not found: ID does not exist" containerID="f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d" Aug 04 15:55:50.815538 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.815496 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d"} err="failed to get container status \"f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d\": rpc error: code = NotFound desc = could not find container \"f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d\": container with ID starting with f7a7c96795eb0231f6c0a3bac7572aaf2c9e5c69110dac04fe503f7333f0733d not found: ID does not exist" Aug 04 15:55:50.828823 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.828800 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z"] Aug 04 15:55:50.830598 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.830564 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-21cd04-cfdb7bdd8-tgk6z"] Aug 04 15:55:50.943601 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:50.943533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:51.288652 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:51.288624 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93bc4239-db30-47a4-8268-faa1ece4652f" path="/var/lib/kubelet/pods/93bc4239-db30-47a4-8268-faa1ece4652f/volumes" Aug 04 15:55:57.021866 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:57.021831 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:57.093258 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:57.093218 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:57.297063 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:57.296986 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:55:57.554865 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:55:57.554780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:08.202270 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.202237 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9"] Aug 04 15:56:08.202896 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.202449 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" podUID="1d2237c4-8b8d-4338-8686-502ebb40c4ff" containerName="maas-api" containerID="cri-o://90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22" gracePeriod=30 Aug 04 15:56:08.235848 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.235812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:08.310315 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.310282 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:08.377099 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.377063 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:08.667140 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.667115 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:56:08.761947 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.761862 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqz24\" (UniqueName: \"kubernetes.io/projected/1d2237c4-8b8d-4338-8686-502ebb40c4ff-kube-api-access-cqz24\") pod \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " Aug 04 15:56:08.761947 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.761929 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2237c4-8b8d-4338-8686-502ebb40c4ff-openshift-service-ca\") pod \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " Aug 04 15:56:08.762145 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.761967 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls\") pod \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\" (UID: \"1d2237c4-8b8d-4338-8686-502ebb40c4ff\") " Aug 04 15:56:08.762401 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.762375 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2237c4-8b8d-4338-8686-502ebb40c4ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d2237c4-8b8d-4338-8686-502ebb40c4ff" (UID: "1d2237c4-8b8d-4338-8686-502ebb40c4ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:56:08.764118 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.764085 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2237c4-8b8d-4338-8686-502ebb40c4ff-kube-api-access-cqz24" (OuterVolumeSpecName: "kube-api-access-cqz24") pod "1d2237c4-8b8d-4338-8686-502ebb40c4ff" (UID: "1d2237c4-8b8d-4338-8686-502ebb40c4ff"). InnerVolumeSpecName "kube-api-access-cqz24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:56:08.764118 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.764107 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d2237c4-8b8d-4338-8686-502ebb40c4ff" (UID: "1d2237c4-8b8d-4338-8686-502ebb40c4ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:56:08.862534 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.862494 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d2237c4-8b8d-4338-8686-502ebb40c4ff-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:56:08.862534 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.862529 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqz24\" (UniqueName: \"kubernetes.io/projected/1d2237c4-8b8d-4338-8686-502ebb40c4ff-kube-api-access-cqz24\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:56:08.862534 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.862539 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d2237c4-8b8d-4338-8686-502ebb40c4ff-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:56:08.894906 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.894872 2559 generic.go:356] "Generic (PLEG): container finished" podID="1d2237c4-8b8d-4338-8686-502ebb40c4ff" containerID="90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22" exitCode=0 Aug 04 15:56:08.895058 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.894918 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" event={"ID":"1d2237c4-8b8d-4338-8686-502ebb40c4ff","Type":"ContainerDied","Data":"90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22"} Aug 04 15:56:08.895058 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.894930 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" Aug 04 15:56:08.895058 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.894985 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9" event={"ID":"1d2237c4-8b8d-4338-8686-502ebb40c4ff","Type":"ContainerDied","Data":"e12a87663de5f5369c1ae6806db01e3e72a2be60987cf1b468e0d308b099fb02"} Aug 04 15:56:08.895058 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.895008 2559 scope.go:117] "RemoveContainer" containerID="90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22" Aug 04 15:56:08.904081 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.904064 2559 scope.go:117] "RemoveContainer" containerID="90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22" Aug 04 15:56:08.904298 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:56:08.904277 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22\": container with ID starting with 90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22 not found: ID does not exist" containerID="90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22" Aug 04 15:56:08.904354 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.904305 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22"} err="failed to get container status \"90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22\": rpc error: code = NotFound desc = could not find container \"90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22\": container with ID starting with 90da00bc971ce18e86a526238115edfe709c2e238e66350533d1c6c4db72ca22 not found: ID does not exist" Aug 04 15:56:08.917137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.917107 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9"] Aug 04 15:56:08.922649 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:08.922627 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9d3b05-787fd7f6b-zwzg9"] Aug 04 15:56:09.289003 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:09.288970 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d2237c4-8b8d-4338-8686-502ebb40c4ff" path="/var/lib/kubelet/pods/1d2237c4-8b8d-4338-8686-502ebb40c4ff/volumes" Aug 04 15:56:11.100345 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:11.100313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:11.200746 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:11.200715 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:14.605372 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:14.605338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:14.673169 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:14.673139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:14.910203 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:14.910121 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:15.129075 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:15.129042 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:15.494651 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:15.494620 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:15.834046 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:15.833946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:16.200170 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:16.200138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:18.960101 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:18.960068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:19.051162 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:19.051130 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:20.040447 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.040411 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn"] Aug 04 15:56:20.041050 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.041031 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d2237c4-8b8d-4338-8686-502ebb40c4ff" containerName="maas-api" Aug 04 15:56:20.041050 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.041051 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d2237c4-8b8d-4338-8686-502ebb40c4ff" containerName="maas-api" Aug 04 15:56:20.041224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.041077 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93bc4239-db30-47a4-8268-faa1ece4652f" containerName="maas-api" Aug 04 15:56:20.041224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.041087 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="93bc4239-db30-47a4-8268-faa1ece4652f" containerName="maas-api" Aug 04 15:56:20.041224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.041176 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="93bc4239-db30-47a4-8268-faa1ece4652f" containerName="maas-api" Aug 04 15:56:20.041224 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.041192 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d2237c4-8b8d-4338-8686-502ebb40c4ff" containerName="maas-api" Aug 04 15:56:20.044785 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.044762 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.047439 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.047414 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ddf7a7-serving-cert\"" Aug 04 15:56:20.053704 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.053680 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn"] Aug 04 15:56:20.120124 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.120086 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:20.163545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.163506 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.163764 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.163604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49aac14-2fd1-4192-bf6d-55053a735d18-openshift-service-ca\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.163764 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.163645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4rd\" (UniqueName: \"kubernetes.io/projected/f49aac14-2fd1-4192-bf6d-55053a735d18-kube-api-access-zg4rd\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.264909 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.264875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49aac14-2fd1-4192-bf6d-55053a735d18-openshift-service-ca\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.265110 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.264926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4rd\" (UniqueName: \"kubernetes.io/projected/f49aac14-2fd1-4192-bf6d-55053a735d18-kube-api-access-zg4rd\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.265110 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.264973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.265110 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:56:20.265087 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-serving-cert: secret "maas-api-e2e-api-a-ddf7a7-serving-cert" not found Aug 04 15:56:20.265279 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:56:20.265166 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls podName:f49aac14-2fd1-4192-bf6d-55053a735d18 nodeName:}" failed. No retries permitted until 2026-08-04 15:56:20.765146598 +0000 UTC m=+2746.119234340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls") pod "maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" (UID: "f49aac14-2fd1-4192-bf6d-55053a735d18") : secret "maas-api-e2e-api-a-ddf7a7-serving-cert" not found Aug 04 15:56:20.265511 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.265490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49aac14-2fd1-4192-bf6d-55053a735d18-openshift-service-ca\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.274986 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.274960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4rd\" (UniqueName: \"kubernetes.io/projected/f49aac14-2fd1-4192-bf6d-55053a735d18-kube-api-access-zg4rd\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.455445 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.455410 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:20.678196 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.678161 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:20.769760 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.769676 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.772262 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.772231 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls\") pod \"maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:20.783771 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.783748 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:20.957504 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:20.957477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:21.098246 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:21.098223 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn"] Aug 04 15:56:21.099983 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:56:21.099956 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49aac14_2fd1_4192_bf6d_55053a735d18.slice/crio-6a5345a429defbd42b43674b9fe4c9a35f8ae0b387d8e03379d29d05ac113b94 WatchSource:0}: Error finding container 6a5345a429defbd42b43674b9fe4c9a35f8ae0b387d8e03379d29d05ac113b94: Status 404 returned error can't find the container with id 6a5345a429defbd42b43674b9fe4c9a35f8ae0b387d8e03379d29d05ac113b94 Aug 04 15:56:21.104189 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:21.104172 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 15:56:21.967119 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:21.967074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" event={"ID":"f49aac14-2fd1-4192-bf6d-55053a735d18","Type":"ContainerStarted","Data":"b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b"} Aug 04 15:56:21.967352 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:21.967148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" event={"ID":"f49aac14-2fd1-4192-bf6d-55053a735d18","Type":"ContainerStarted","Data":"6a5345a429defbd42b43674b9fe4c9a35f8ae0b387d8e03379d29d05ac113b94"} Aug 04 15:56:21.967352 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:21.967187 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:21.990324 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:21.990266 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" podStartSLOduration=1.457746449 podStartE2EDuration="1.990249557s" totalImagesPullingTime="532.503108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:21.989563231 +0000 UTC m=+2747.343650993" watchObservedRunningTime="2026-08-04 15:56:21.990249557 +0000 UTC m=+2747.344337317" Aug 04 15:56:27.977856 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:27.977820 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:56:42.449114 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:42.449082 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:42.546061 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:42.546028 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:44.009809 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:44.009779 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:44.496791 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:44.496743 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:44.815823 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:44.815739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:45.063479 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:45.063442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:48.034815 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:48.034739 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:48.131401 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:48.131366 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:49.220295 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.220262 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv"] Aug 04 15:56:49.224434 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.224417 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.227411 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.227359 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-063515-serving-cert\"" Aug 04 15:56:49.249010 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.248988 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv"] Aug 04 15:56:49.278885 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.278853 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:49.318546 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.318510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-maas-api-tls\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.318767 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.318556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.318767 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.318751 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rz8\" (UniqueName: \"kubernetes.io/projected/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-kube-api-access-88rz8\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.419993 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.419952 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-maas-api-tls\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.419993 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.419997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.420253 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.420074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rz8\" (UniqueName: \"kubernetes.io/projected/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-kube-api-access-88rz8\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.420771 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.420749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-openshift-service-ca\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.422437 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.422414 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-maas-api-tls\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.436839 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.436809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rz8\" (UniqueName: \"kubernetes.io/projected/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-kube-api-access-88rz8\") pod \"maas-api-e2e-api-b-063515-665c8d64f6-f9dlv\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.473289 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.473213 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:49.536229 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.536203 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:49.682178 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.682149 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv"] Aug 04 15:56:49.683960 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:56:49.683934 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fccd43_c9fd_4f0f_9b90_5c5ce735e1b0.slice/crio-4a2f2dc5dde0f8a443c51a3094d3b600afc5a5810ef00dd049553240db26b3fa WatchSource:0}: Error finding container 4a2f2dc5dde0f8a443c51a3094d3b600afc5a5810ef00dd049553240db26b3fa: Status 404 returned error can't find the container with id 4a2f2dc5dde0f8a443c51a3094d3b600afc5a5810ef00dd049553240db26b3fa Aug 04 15:56:49.956282 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:49.956241 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:50.056889 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:50.056857 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:56:50.117856 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:50.117817 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" event={"ID":"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0","Type":"ContainerStarted","Data":"4a2f2dc5dde0f8a443c51a3094d3b600afc5a5810ef00dd049553240db26b3fa"} Aug 04 15:56:51.125081 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:51.125046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" event={"ID":"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0","Type":"ContainerStarted","Data":"ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d"} Aug 04 15:56:51.125534 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:51.125127 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:56:51.156405 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:51.156340 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" podStartSLOduration=1.620106833 podStartE2EDuration="2.156317195s" totalImagesPullingTime="536.210362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:56:51.152860114 +0000 UTC m=+2776.506947873" watchObservedRunningTime="2026-08-04 15:56:51.156317195 +0000 UTC m=+2776.510404954" Aug 04 15:56:57.135512 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:56:57.135482 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:57:13.134873 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:13.134839 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:31.477436 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.477401 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn"] Aug 04 15:57:31.479888 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.477646 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" podUID="f49aac14-2fd1-4192-bf6d-55053a735d18" containerName="maas-api" containerID="cri-o://b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b" gracePeriod=30 Aug 04 15:57:31.509319 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.509285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:31.631272 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.631239 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:31.730983 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.730866 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:31.737136 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.737109 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:57:31.788169 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.788135 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49aac14-2fd1-4192-bf6d-55053a735d18-openshift-service-ca\") pod \"f49aac14-2fd1-4192-bf6d-55053a735d18\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " Aug 04 15:57:31.788336 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.788176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg4rd\" (UniqueName: \"kubernetes.io/projected/f49aac14-2fd1-4192-bf6d-55053a735d18-kube-api-access-zg4rd\") pod \"f49aac14-2fd1-4192-bf6d-55053a735d18\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " Aug 04 15:57:31.788336 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.788244 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls\") pod \"f49aac14-2fd1-4192-bf6d-55053a735d18\" (UID: \"f49aac14-2fd1-4192-bf6d-55053a735d18\") " Aug 04 15:57:31.788622 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.788595 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f49aac14-2fd1-4192-bf6d-55053a735d18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f49aac14-2fd1-4192-bf6d-55053a735d18" (UID: "f49aac14-2fd1-4192-bf6d-55053a735d18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:57:31.790379 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.790356 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f49aac14-2fd1-4192-bf6d-55053a735d18" (UID: "f49aac14-2fd1-4192-bf6d-55053a735d18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:57:31.790470 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.790443 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f49aac14-2fd1-4192-bf6d-55053a735d18-kube-api-access-zg4rd" (OuterVolumeSpecName: "kube-api-access-zg4rd") pod "f49aac14-2fd1-4192-bf6d-55053a735d18" (UID: "f49aac14-2fd1-4192-bf6d-55053a735d18"). InnerVolumeSpecName "kube-api-access-zg4rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:57:31.889658 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.889627 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f49aac14-2fd1-4192-bf6d-55053a735d18-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:57:31.889658 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.889656 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg4rd\" (UniqueName: \"kubernetes.io/projected/f49aac14-2fd1-4192-bf6d-55053a735d18-kube-api-access-zg4rd\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:57:31.889658 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:31.889666 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f49aac14-2fd1-4192-bf6d-55053a735d18-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:57:32.131978 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.131852 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:32.334298 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.334266 2559 generic.go:356] "Generic (PLEG): container finished" podID="f49aac14-2fd1-4192-bf6d-55053a735d18" containerID="b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b" exitCode=0 Aug 04 15:57:32.334476 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.334314 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" Aug 04 15:57:32.334476 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.334325 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" event={"ID":"f49aac14-2fd1-4192-bf6d-55053a735d18","Type":"ContainerDied","Data":"b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b"} Aug 04 15:57:32.334476 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.334368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn" event={"ID":"f49aac14-2fd1-4192-bf6d-55053a735d18","Type":"ContainerDied","Data":"6a5345a429defbd42b43674b9fe4c9a35f8ae0b387d8e03379d29d05ac113b94"} Aug 04 15:57:32.334476 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.334389 2559 scope.go:117] "RemoveContainer" containerID="b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b" Aug 04 15:57:32.343787 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.343767 2559 scope.go:117] "RemoveContainer" containerID="b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b" Aug 04 15:57:32.344087 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:57:32.344068 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b\": container with ID starting with b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b not found: ID does not exist" containerID="b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b" Aug 04 15:57:32.344175 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.344100 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b"} err="failed to get container status \"b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b\": rpc error: code = NotFound desc = could not find container \"b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b\": container with ID starting with b653d6369ff7a708da132c61ad022e0353c852f89d79655a34d491507d67c49b not found: ID does not exist" Aug 04 15:57:32.360260 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.360232 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn"] Aug 04 15:57:32.361999 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:32.361963 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ddf7a7-5765cb8cb8-kkwmn"] Aug 04 15:57:33.289245 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:33.289212 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f49aac14-2fd1-4192-bf6d-55053a735d18" path="/var/lib/kubelet/pods/f49aac14-2fd1-4192-bf6d-55053a735d18/volumes" Aug 04 15:57:38.456590 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:38.456532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:38.567314 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:38.567280 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:38.772511 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:38.772424 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:38.977444 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:38.977406 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:54.603888 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.603853 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv"] Aug 04 15:57:54.604297 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.604118 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" podUID="b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" containerName="maas-api" containerID="cri-o://ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d" gracePeriod=30 Aug 04 15:57:54.637206 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.637138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:54.716003 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.715964 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:54.816863 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.816823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:54.869621 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.869599 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:57:54.998375 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.998337 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-openshift-service-ca\") pod \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " Aug 04 15:57:54.998544 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.998405 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88rz8\" (UniqueName: \"kubernetes.io/projected/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-kube-api-access-88rz8\") pod \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " Aug 04 15:57:54.998544 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.998465 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-maas-api-tls\") pod \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\" (UID: \"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0\") " Aug 04 15:57:54.998777 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:54.998750 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" (UID: "b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 15:57:55.000558 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.000535 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-kube-api-access-88rz8" (OuterVolumeSpecName: "kube-api-access-88rz8") pod "b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" (UID: "b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0"). InnerVolumeSpecName "kube-api-access-88rz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 15:57:55.000729 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.000565 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" (UID: "b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 15:57:55.099504 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.099467 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:57:55.099504 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.099500 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:57:55.099504 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.099510 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88rz8\" (UniqueName: \"kubernetes.io/projected/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0-kube-api-access-88rz8\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 15:57:55.203408 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.203380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:57:55.451236 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.451203 2559 generic.go:356] "Generic (PLEG): container finished" podID="b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" containerID="ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d" exitCode=0 Aug 04 15:57:55.451395 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.451251 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" Aug 04 15:57:55.451395 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.451278 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" event={"ID":"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0","Type":"ContainerDied","Data":"ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d"} Aug 04 15:57:55.451395 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.451336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv" event={"ID":"b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0","Type":"ContainerDied","Data":"4a2f2dc5dde0f8a443c51a3094d3b600afc5a5810ef00dd049553240db26b3fa"} Aug 04 15:57:55.451395 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.451365 2559 scope.go:117] "RemoveContainer" containerID="ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d" Aug 04 15:57:55.460865 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.460845 2559 scope.go:117] "RemoveContainer" containerID="ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d" Aug 04 15:57:55.461173 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:57:55.461148 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d\": container with ID starting with ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d not found: ID does not exist" containerID="ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d" Aug 04 15:57:55.461273 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.461185 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d"} err="failed to get container status \"ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d\": rpc error: code = NotFound desc = could not find container \"ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d\": container with ID starting with ba31cb3d4c43745ca606cba44054e9a9cf9d0bd31173d939389887f49430e80d not found: ID does not exist" Aug 04 15:57:55.478249 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.478223 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv"] Aug 04 15:57:55.484100 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:55.484077 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-063515-665c8d64f6-f9dlv"] Aug 04 15:57:57.288467 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:57:57.288431 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" path="/var/lib/kubelet/pods/b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0/volumes" Aug 04 15:58:02.016125 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:02.016092 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:02.114815 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:02.114783 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:02.315206 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:02.315125 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:02.546173 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:02.546138 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:02.930222 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:02.930192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:03.247474 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:03.247443 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:03.522278 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:03.522197 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:06.505206 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:06.505172 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:06.607539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:06.607503 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:07.723214 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.723177 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8"] Aug 04 15:58:07.723797 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.723781 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" containerName="maas-api" Aug 04 15:58:07.723859 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.723800 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" containerName="maas-api" Aug 04 15:58:07.723859 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.723814 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f49aac14-2fd1-4192-bf6d-55053a735d18" containerName="maas-api" Aug 04 15:58:07.723859 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.723823 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f49aac14-2fd1-4192-bf6d-55053a735d18" containerName="maas-api" Aug 04 15:58:07.723960 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.723935 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f49aac14-2fd1-4192-bf6d-55053a735d18" containerName="maas-api" Aug 04 15:58:07.723960 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.723948 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1fccd43-c9fd-4f0f-9b90-5c5ce735e1b0" containerName="maas-api" Aug 04 15:58:07.728701 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.728675 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.734348 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.734326 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-67a15f-serving-cert\"" Aug 04 15:58:07.739350 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.739313 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8"] Aug 04 15:58:07.762516 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.762485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:07.807861 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.807824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xwh\" (UniqueName: \"kubernetes.io/projected/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-kube-api-access-v2xwh\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.808007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.807879 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.808007 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.807947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.909068 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.909026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xwh\" (UniqueName: \"kubernetes.io/projected/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-kube-api-access-v2xwh\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.909259 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.909078 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.909259 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.909107 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.909259 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:58:07.909200 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-serving-cert: secret "maas-api-e2e-shared-a-67a15f-serving-cert" not found Aug 04 15:58:07.909393 ip-10-0-138-35 kubenswrapper[2559]: E0804 15:58:07.909264 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls podName:294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9 nodeName:}" failed. No retries permitted until 2026-08-04 15:58:08.409247881 +0000 UTC m=+2853.763335618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls") pod "maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" (UID: "294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9") : secret "maas-api-e2e-shared-a-67a15f-serving-cert" not found Aug 04 15:58:07.909819 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.909799 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:07.961337 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:07.961310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xwh\" (UniqueName: \"kubernetes.io/projected/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-kube-api-access-v2xwh\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:08.197089 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:08.197057 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:08.412736 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:08.412706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:08.415492 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:08.415463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:08.640645 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:08.640553 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:08.779388 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:08.779345 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8"] Aug 04 15:58:08.783481 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:58:08.783443 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294a5f48_4e0a_4dc9_bad5_6ea47c79c6f9.slice/crio-68c17838aa8ca4eba255551e66da9785d1318fbb5175e5a194b6e1645de909bd WatchSource:0}: Error finding container 68c17838aa8ca4eba255551e66da9785d1318fbb5175e5a194b6e1645de909bd: Status 404 returned error can't find the container with id 68c17838aa8ca4eba255551e66da9785d1318fbb5175e5a194b6e1645de909bd Aug 04 15:58:09.324558 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:09.324529 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:09.423627 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:09.423601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:09.527349 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:09.527306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" event={"ID":"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9","Type":"ContainerStarted","Data":"342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f"} Aug 04 15:58:09.527545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:09.527360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" event={"ID":"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9","Type":"ContainerStarted","Data":"68c17838aa8ca4eba255551e66da9785d1318fbb5175e5a194b6e1645de909bd"} Aug 04 15:58:09.527545 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:09.527431 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:09.548066 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:09.548014 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" podStartSLOduration=1.927827248 podStartE2EDuration="2.547996951s" totalImagesPullingTime="620.169703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:58:09.545850728 +0000 UTC m=+2854.899938490" watchObservedRunningTime="2026-08-04 15:58:09.547996951 +0000 UTC m=+2854.902084717" Aug 04 15:58:15.538501 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:15.538465 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 15:58:31.013781 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:31.013693 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:31.499384 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:31.499319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:31.852930 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:31.852806 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:32.202056 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:32.202014 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:32.951118 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:32.951086 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:34.690520 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:34.690488 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:34.783219 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:34.783185 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:35.764646 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.764525 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c"] Aug 04 15:58:35.783338 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.783297 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c"] Aug 04 15:58:35.783513 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.783417 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.786157 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.786130 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4d9499-serving-cert\"" Aug 04 15:58:35.814086 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.814054 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:35.867539 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.867508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.867725 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.867546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbp2\" (UniqueName: \"kubernetes.io/projected/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-kube-api-access-lrbp2\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.867725 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.867643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-maas-api-tls\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.897913 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.897882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:35.968096 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.968062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.968096 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.968097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbp2\" (UniqueName: \"kubernetes.io/projected/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-kube-api-access-lrbp2\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.968366 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.968148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-maas-api-tls\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.968726 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.968703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.970632 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.970606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-maas-api-tls\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:35.977238 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:35.977212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbp2\" (UniqueName: \"kubernetes.io/projected/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-kube-api-access-lrbp2\") pod \"maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:36.094237 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:36.094148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:36.241272 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:36.241246 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c"] Aug 04 15:58:36.242464 ip-10-0-138-35 kubenswrapper[2559]: W0804 15:58:36.242439 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cbaa90_8ed8_4a9d_9ca0_cb332fc5260f.slice/crio-4725e630269833402d002c22b9b3b094db2cb3838e86de055cac5c0fbfe25eca WatchSource:0}: Error finding container 4725e630269833402d002c22b9b3b094db2cb3838e86de055cac5c0fbfe25eca: Status 404 returned error can't find the container with id 4725e630269833402d002c22b9b3b094db2cb3838e86de055cac5c0fbfe25eca Aug 04 15:58:36.665173 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:36.665133 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" event={"ID":"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f","Type":"ContainerStarted","Data":"4725e630269833402d002c22b9b3b094db2cb3838e86de055cac5c0fbfe25eca"} Aug 04 15:58:36.804971 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:36.804938 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:36.903426 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:36.903349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:37.679732 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:37.679692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" event={"ID":"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f","Type":"ContainerStarted","Data":"33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429"} Aug 04 15:58:37.679952 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:37.679816 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:37.720847 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:37.720795 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" podStartSLOduration=2.288753847 podStartE2EDuration="2.720779032s" totalImagesPullingTime="432.025185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 15:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 15:58:37.719853477 +0000 UTC m=+2883.073941235" watchObservedRunningTime="2026-08-04 15:58:37.720779032 +0000 UTC m=+2883.074866791" Aug 04 15:58:43.691160 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:43.691132 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 15:58:46.374497 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:46.374465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:58:59.827732 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:58:59.827694 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:03.730380 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:59:03.730341 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 15:59:52.252137 ip-10-0-138-35 kubenswrapper[2559]: I0804 15:59:52.252104 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:28.764695 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:00:28.764606 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:00:35.354378 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:00:35.354351 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 16:00:35.359274 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:00:35.359251 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 16:00:35.364882 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:00:35.364864 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 16:00:35.369299 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:00:35.369283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 16:02:26.436213 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:02:26.436182 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:11.072111 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:04:11.072076 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:11.805207 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:04:11.805171 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:22.639643 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:04:22.639597 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:26.933463 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:04:26.933428 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:04:40.904372 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:04:40.904288 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:02.372499 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:05:02.372468 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:05:35.384652 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:05:35.384627 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 16:05:35.389199 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:05:35.389174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 16:05:35.396666 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:05:35.396648 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 16:05:35.400693 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:05:35.400673 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 16:05:57.494429 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:05:57.494347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:34.581948 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:34.581909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:35.480482 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:35.480445 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:45.707419 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.707386 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h"] Aug 04 16:06:45.710829 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.710813 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.714194 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.714173 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67a15f\"/\"default-dockercfg-zbw8p\"" Aug 04 16:06:45.715221 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.715204 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67a15f\"/\"openshift-service-ca.crt\"" Aug 04 16:06:45.715323 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.715216 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-67a15f\"/\"kube-root-ca.crt\"" Aug 04 16:06:45.715323 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.715269 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-67a15f\"/\"rate-test-model-67a15f-kserve-self-signed-certs\"" Aug 04 16:06:45.724026 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.724000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h"] Aug 04 16:06:45.749958 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.749931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-tmp-dir\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.750073 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.749966 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca90a549-95ff-4626-aa81-435829eb3ce4-tls-certs\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.750073 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.749994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-home\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.750073 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.750025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-model-cache\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.750200 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.750082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-dshm\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.750200 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.750138 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26sl6\" (UniqueName: \"kubernetes.io/projected/ca90a549-95ff-4626-aa81-435829eb3ce4-kube-api-access-26sl6\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.753039 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.753018 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:06:45.850894 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.850857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-model-cache\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851090 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.850899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-dshm\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851090 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.850919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26sl6\" (UniqueName: \"kubernetes.io/projected/ca90a549-95ff-4626-aa81-435829eb3ce4-kube-api-access-26sl6\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851090 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.851009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-tmp-dir\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851090 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.851042 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca90a549-95ff-4626-aa81-435829eb3ce4-tls-certs\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851090 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.851068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-home\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851434 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.851398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-model-cache\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851489 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.851444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-tmp-dir\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.851489 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.851485 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-home\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.853173 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.853152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-dshm\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.853511 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.853493 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca90a549-95ff-4626-aa81-435829eb3ce4-tls-certs\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:45.859654 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:45.859633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26sl6\" (UniqueName: \"kubernetes.io/projected/ca90a549-95ff-4626-aa81-435829eb3ce4-kube-api-access-26sl6\") pod \"rate-test-model-67a15f-kserve-5988bbc979-86g6h\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:46.019793 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:46.019713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:46.159847 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:46.159821 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h"] Aug 04 16:06:46.162007 ip-10-0-138-35 kubenswrapper[2559]: W0804 16:06:46.161967 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca90a549_95ff_4626_aa81_435829eb3ce4.slice/crio-9b27907ec196784a8597a9e6aa1c028ba8c070435d6ba17b5364b5ed0239322d WatchSource:0}: Error finding container 9b27907ec196784a8597a9e6aa1c028ba8c070435d6ba17b5364b5ed0239322d: Status 404 returned error can't find the container with id 9b27907ec196784a8597a9e6aa1c028ba8c070435d6ba17b5364b5ed0239322d Aug 04 16:06:46.174509 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:46.174486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" event={"ID":"ca90a549-95ff-4626-aa81-435829eb3ce4","Type":"ContainerStarted","Data":"9b27907ec196784a8597a9e6aa1c028ba8c070435d6ba17b5364b5ed0239322d"} Aug 04 16:06:47.181141 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:47.181099 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" event={"ID":"ca90a549-95ff-4626-aa81-435829eb3ce4","Type":"ContainerStarted","Data":"17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5"} Aug 04 16:06:47.203369 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:47.203311 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" podStartSLOduration=2.203291853 podStartE2EDuration="2.203291853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:06:47.202460135 +0000 UTC m=+3372.556547893" watchObservedRunningTime="2026-08-04 16:06:47.203291853 +0000 UTC m=+3372.557379613" Aug 04 16:06:56.019845 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:56.019813 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:56.020405 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:56.019870 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:56.033114 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:56.033092 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:06:56.237355 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:06:56.237328 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:07:34.848856 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:07:34.848774 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:04.932691 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:04.932609 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h"] Aug 04 16:09:05.010770 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:05.010730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:05.871292 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:05.871256 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:19.951513 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:19.951466 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" podUID="ca90a549-95ff-4626-aa81-435829eb3ce4" containerName="main" containerID="cri-o://17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5" gracePeriod=45 Aug 04 16:09:20.200398 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.200373 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:09:20.257014 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.256941 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26sl6\" (UniqueName: \"kubernetes.io/projected/ca90a549-95ff-4626-aa81-435829eb3ce4-kube-api-access-26sl6\") pod \"ca90a549-95ff-4626-aa81-435829eb3ce4\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " Aug 04 16:09:20.257175 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257031 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-dshm\") pod \"ca90a549-95ff-4626-aa81-435829eb3ce4\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " Aug 04 16:09:20.257175 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257073 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-home\") pod \"ca90a549-95ff-4626-aa81-435829eb3ce4\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " Aug 04 16:09:20.257175 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257118 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-model-cache\") pod \"ca90a549-95ff-4626-aa81-435829eb3ce4\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " Aug 04 16:09:20.257175 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257165 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-tmp-dir\") pod \"ca90a549-95ff-4626-aa81-435829eb3ce4\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " Aug 04 16:09:20.257378 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257202 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca90a549-95ff-4626-aa81-435829eb3ce4-tls-certs\") pod \"ca90a549-95ff-4626-aa81-435829eb3ce4\" (UID: \"ca90a549-95ff-4626-aa81-435829eb3ce4\") " Aug 04 16:09:20.257433 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257394 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-home" (OuterVolumeSpecName: "home") pod "ca90a549-95ff-4626-aa81-435829eb3ce4" (UID: "ca90a549-95ff-4626-aa81-435829eb3ce4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:09:20.257433 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca90a549-95ff-4626-aa81-435829eb3ce4" (UID: "ca90a549-95ff-4626-aa81-435829eb3ce4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:09:20.257433 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257409 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca90a549-95ff-4626-aa81-435829eb3ce4" (UID: "ca90a549-95ff-4626-aa81-435829eb3ce4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:09:20.257554 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257497 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-home\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:09:20.257554 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257510 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-model-cache\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:09:20.257554 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.257520 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-tmp-dir\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:09:20.259358 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.259338 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-dshm" (OuterVolumeSpecName: "dshm") pod "ca90a549-95ff-4626-aa81-435829eb3ce4" (UID: "ca90a549-95ff-4626-aa81-435829eb3ce4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Aug 04 16:09:20.259358 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.259346 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca90a549-95ff-4626-aa81-435829eb3ce4-kube-api-access-26sl6" (OuterVolumeSpecName: "kube-api-access-26sl6") pod "ca90a549-95ff-4626-aa81-435829eb3ce4" (UID: "ca90a549-95ff-4626-aa81-435829eb3ce4"). InnerVolumeSpecName "kube-api-access-26sl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:09:20.259464 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.259355 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca90a549-95ff-4626-aa81-435829eb3ce4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca90a549-95ff-4626-aa81-435829eb3ce4" (UID: "ca90a549-95ff-4626-aa81-435829eb3ce4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:09:20.358950 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.358919 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca90a549-95ff-4626-aa81-435829eb3ce4-dshm\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:09:20.358950 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.358948 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca90a549-95ff-4626-aa81-435829eb3ce4-tls-certs\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:09:20.358950 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.358958 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26sl6\" (UniqueName: \"kubernetes.io/projected/ca90a549-95ff-4626-aa81-435829eb3ce4-kube-api-access-26sl6\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:09:20.974084 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.974045 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca90a549-95ff-4626-aa81-435829eb3ce4" containerID="17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5" exitCode=0 Aug 04 16:09:20.974476 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.974092 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" Aug 04 16:09:20.974476 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.974125 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" event={"ID":"ca90a549-95ff-4626-aa81-435829eb3ce4","Type":"ContainerDied","Data":"17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5"} Aug 04 16:09:20.974476 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.974166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h" event={"ID":"ca90a549-95ff-4626-aa81-435829eb3ce4","Type":"ContainerDied","Data":"9b27907ec196784a8597a9e6aa1c028ba8c070435d6ba17b5364b5ed0239322d"} Aug 04 16:09:20.974476 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.974182 2559 scope.go:117] "RemoveContainer" containerID="17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5" Aug 04 16:09:20.983491 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.983473 2559 scope.go:117] "RemoveContainer" containerID="17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5" Aug 04 16:09:20.983757 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:09:20.983738 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5\": container with ID starting with 17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5 not found: ID does not exist" containerID="17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5" Aug 04 16:09:20.983816 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.983765 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5"} err="failed to get container status \"17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5\": rpc error: code = NotFound desc = could not find container \"17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5\": container with ID starting with 17f03c91c43ca4439c4689c287d064b0bfcd24bec7f2bb7c81129f09bb6d66a5 not found: ID does not exist" Aug 04 16:09:20.996567 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:20.996538 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h"] Aug 04 16:09:21.000909 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:21.000890 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-67a15f/rate-test-model-67a15f-kserve-5988bbc979-86g6h"] Aug 04 16:09:21.289006 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:21.288929 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca90a549-95ff-4626-aa81-435829eb3ce4" path="/var/lib/kubelet/pods/ca90a549-95ff-4626-aa81-435829eb3ce4/volumes" Aug 04 16:09:25.460755 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:25.460710 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:25.890773 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:25.890685 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:26.199346 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:26.199315 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:26.716624 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:26.716557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:26.817273 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:26.817237 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:29.343917 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:29.343877 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:29.474757 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:29.474713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:30.457047 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.457013 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst"] Aug 04 16:09:30.457699 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.457676 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca90a549-95ff-4626-aa81-435829eb3ce4" containerName="main" Aug 04 16:09:30.457699 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.457699 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca90a549-95ff-4626-aa81-435829eb3ce4" containerName="main" Aug 04 16:09:30.457878 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.457828 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca90a549-95ff-4626-aa81-435829eb3ce4" containerName="main" Aug 04 16:09:30.462181 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.462161 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.465770 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.465746 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-8ff9cf-serving-cert\"" Aug 04 16:09:30.477905 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.477883 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst"] Aug 04 16:09:30.506147 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.506118 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:30.549933 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.549900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qwj\" (UniqueName: \"kubernetes.io/projected/004bd657-3055-4ccb-b5d9-749cf5d64e0c-kube-api-access-w6qwj\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.550076 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.549948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004bd657-3055-4ccb-b5d9-749cf5d64e0c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.550076 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.549995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004bd657-3055-4ccb-b5d9-749cf5d64e0c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.647308 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.647274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:30.651320 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.651296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qwj\" (UniqueName: \"kubernetes.io/projected/004bd657-3055-4ccb-b5d9-749cf5d64e0c-kube-api-access-w6qwj\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.651400 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.651343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004bd657-3055-4ccb-b5d9-749cf5d64e0c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.651447 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.651400 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004bd657-3055-4ccb-b5d9-749cf5d64e0c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.652051 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.652030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004bd657-3055-4ccb-b5d9-749cf5d64e0c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.653714 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.653695 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004bd657-3055-4ccb-b5d9-749cf5d64e0c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.661967 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.661941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qwj\" (UniqueName: \"kubernetes.io/projected/004bd657-3055-4ccb-b5d9-749cf5d64e0c-kube-api-access-w6qwj\") pod \"maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.772513 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.772435 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:30.918949 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.918919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst"] Aug 04 16:09:30.919830 ip-10-0-138-35 kubenswrapper[2559]: W0804 16:09:30.919796 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004bd657_3055_4ccb_b5d9_749cf5d64e0c.slice/crio-069a66f79fe4c4c91685ca45c50fc105ccf85c8f303a8411c332abcde69acd56 WatchSource:0}: Error finding container 069a66f79fe4c4c91685ca45c50fc105ccf85c8f303a8411c332abcde69acd56: Status 404 returned error can't find the container with id 069a66f79fe4c4c91685ca45c50fc105ccf85c8f303a8411c332abcde69acd56 Aug 04 16:09:30.921295 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:30.921276 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Aug 04 16:09:31.028040 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:31.027958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" event={"ID":"004bd657-3055-4ccb-b5d9-749cf5d64e0c","Type":"ContainerStarted","Data":"069a66f79fe4c4c91685ca45c50fc105ccf85c8f303a8411c332abcde69acd56"} Aug 04 16:09:31.292148 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:31.292054 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:31.508946 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:31.508913 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:32.035159 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:32.035124 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" event={"ID":"004bd657-3055-4ccb-b5d9-749cf5d64e0c","Type":"ContainerStarted","Data":"2c73301aa2cf042cf78de66799b469858a75fcd332c141164afb4c995749c853"} Aug 04 16:09:32.035329 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:32.035209 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:32.067237 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:32.067182 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" podStartSLOduration=1.33198323 podStartE2EDuration="2.067164496s" totalImagesPullingTime="735.181266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:09:32.065487921 +0000 UTC m=+3537.419575698" watchObservedRunningTime="2026-08-04 16:09:32.067164496 +0000 UTC m=+3537.421252254" Aug 04 16:09:36.233634 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:36.233530 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:38.045853 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:38.045826 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:09:54.384947 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:54.384909 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:55.050234 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:55.050202 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:55.414844 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:55.414755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:55.879160 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:55.879129 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:58.519409 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:58.519373 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:58.620187 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:58.620146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:59.604161 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.604127 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q"] Aug 04 16:09:59.607767 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.607748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.614235 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.614203 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2b50f0-serving-cert\"" Aug 04 16:09:59.617728 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.617700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.617841 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.617774 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.617906 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.617868 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxv4s\" (UniqueName: \"kubernetes.io/projected/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-kube-api-access-nxv4s\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.625050 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.625026 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q"] Aug 04 16:09:59.672328 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.672291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:09:59.718569 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.718532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxv4s\" (UniqueName: \"kubernetes.io/projected/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-kube-api-access-nxv4s\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.718762 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.718621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.718762 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.718681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.718762 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:09:59.718751 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-serving-cert: secret "maas-api-e2e-ipp-b-2b50f0-serving-cert" not found Aug 04 16:09:59.718954 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:09:59.718822 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls podName:3c3cf99b-1a93-45fb-b3f9-942c2ad31784 nodeName:}" failed. No retries permitted until 2026-08-04 16:10:00.218800605 +0000 UTC m=+3565.572888356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls") pod "maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" (UID: "3c3cf99b-1a93-45fb-b3f9-942c2ad31784") : secret "maas-api-e2e-ipp-b-2b50f0-serving-cert" not found Aug 04 16:09:59.719307 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.719286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:09:59.757883 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:09:59.757849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxv4s\" (UniqueName: \"kubernetes.io/projected/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-kube-api-access-nxv4s\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:10:00.223075 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:00.223043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:10:00.225496 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:00.225472 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:10:00.251010 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:00.250958 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:00.398230 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:00.398190 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:00.518711 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:00.518632 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:10:00.689316 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:00.689282 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q"] Aug 04 16:10:00.694697 ip-10-0-138-35 kubenswrapper[2559]: W0804 16:10:00.694671 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3cf99b_1a93_45fb_b3f9_942c2ad31784.slice/crio-63a98f26bd83244227f03e62568a714795731a2f52a5af289d72bfc15b96755e WatchSource:0}: Error finding container 63a98f26bd83244227f03e62568a714795731a2f52a5af289d72bfc15b96755e: Status 404 returned error can't find the container with id 63a98f26bd83244227f03e62568a714795731a2f52a5af289d72bfc15b96755e Aug 04 16:10:01.183658 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:01.183624 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" event={"ID":"3c3cf99b-1a93-45fb-b3f9-942c2ad31784","Type":"ContainerStarted","Data":"63a98f26bd83244227f03e62568a714795731a2f52a5af289d72bfc15b96755e"} Aug 04 16:10:02.190215 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:02.190185 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" event={"ID":"3c3cf99b-1a93-45fb-b3f9-942c2ad31784","Type":"ContainerStarted","Data":"0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3"} Aug 04 16:10:02.190623 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:02.190241 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:10:02.209278 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:02.209221 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" podStartSLOduration=2.572659575 podStartE2EDuration="3.209205605s" totalImagesPullingTime="636.54603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:10:02.207760671 +0000 UTC m=+3567.561848430" watchObservedRunningTime="2026-08-04 16:10:02.209205605 +0000 UTC m=+3567.563293401" Aug 04 16:10:08.200627 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:08.200597 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:10:08.459156 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:08.459079 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:08.557662 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:08.557627 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:27.102008 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:27.101969 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:10:30.467865 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:30.467832 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pzfkj_3494aa11-cbd1-4c2f-bae2-b565ef4d1f80/payload-processing/0.log" Aug 04 16:10:35.414142 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:35.414113 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 16:10:35.418545 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:35.418522 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 16:10:35.427927 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:35.427909 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 16:10:35.432485 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:35.432467 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646lg_194c9b51-0bac-4104-8404-0437944a00f4/ovn-acl-logging/0.log" Aug 04 16:10:45.390316 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:10:45.390285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:26.678928 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:26.678892 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pzfkj_3494aa11-cbd1-4c2f-bae2-b565ef4d1f80/payload-processing/0.log" Aug 04 16:11:27.109850 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:27.109809 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:27.422810 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:27.422729 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:27.853632 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:27.853571 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:30.020013 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:30.019963 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:30.115863 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:30.115824 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:31.117349 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.117317 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4"] Aug 04 16:11:31.121286 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.121262 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.124087 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.124069 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8528c9-serving-cert\"" Aug 04 16:11:31.134314 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.134293 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4"] Aug 04 16:11:31.137650 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.137625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.137752 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.137680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8cm6\" (UniqueName: \"kubernetes.io/projected/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-kube-api-access-x8cm6\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.137839 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.137824 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.165995 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.165945 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:31.238832 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.238787 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8cm6\" (UniqueName: \"kubernetes.io/projected/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-kube-api-access-x8cm6\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.239050 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.238938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.239050 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.238966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.239166 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:11:31.239085 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-serving-cert: secret "maas-api-e2e-ipp-cleanup-8528c9-serving-cert" not found Aug 04 16:11:31.239166 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:11:31.239160 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls podName:202f08dd-aaeb-4076-a61a-47e95ba9c1a9 nodeName:}" failed. No retries permitted until 2026-08-04 16:11:31.739144004 +0000 UTC m=+3657.093231741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" (UID: "202f08dd-aaeb-4076-a61a-47e95ba9c1a9") : secret "maas-api-e2e-ipp-cleanup-8528c9-serving-cert" not found Aug 04 16:11:31.239633 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.239616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.257023 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.256987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8cm6\" (UniqueName: \"kubernetes.io/projected/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-kube-api-access-x8cm6\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.402252 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.402150 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:31.742645 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.742614 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.745009 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.744987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:31.763673 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.763641 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:31.861882 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:31.861847 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:11:32.032891 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:32.032786 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:32.182999 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:32.182976 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4"] Aug 04 16:11:32.185074 ip-10-0-138-35 kubenswrapper[2559]: W0804 16:11:32.185046 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202f08dd_aaeb_4076_a61a_47e95ba9c1a9.slice/crio-4944da2e834d7a9898f6d091e8d1bd17bb2b8dbc38b61736ba1afe5c0eeaa5b4 WatchSource:0}: Error finding container 4944da2e834d7a9898f6d091e8d1bd17bb2b8dbc38b61736ba1afe5c0eeaa5b4: Status 404 returned error can't find the container with id 4944da2e834d7a9898f6d091e8d1bd17bb2b8dbc38b61736ba1afe5c0eeaa5b4 Aug 04 16:11:32.662190 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:32.662157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" event={"ID":"202f08dd-aaeb-4076-a61a-47e95ba9c1a9","Type":"ContainerStarted","Data":"4944da2e834d7a9898f6d091e8d1bd17bb2b8dbc38b61736ba1afe5c0eeaa5b4"} Aug 04 16:11:33.670291 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:33.670255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" event={"ID":"202f08dd-aaeb-4076-a61a-47e95ba9c1a9","Type":"ContainerStarted","Data":"e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345"} Aug 04 16:11:33.670721 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:33.670351 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:33.689104 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:33.689027 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" podStartSLOduration=2.132376581 podStartE2EDuration="2.689007955s" totalImagesPullingTime="556.631374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 16:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:11:33.688922243 +0000 UTC m=+3659.043010003" watchObservedRunningTime="2026-08-04 16:11:33.689007955 +0000 UTC m=+3659.043095728" Aug 04 16:11:39.681152 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:39.681114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:11:55.050394 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:11:55.050358 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:01.831018 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:01.830934 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4"] Aug 04 16:12:01.832271 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:01.831216 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" podUID="202f08dd-aaeb-4076-a61a-47e95ba9c1a9" containerName="maas-api" containerID="cri-o://e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345" gracePeriod=30 Aug 04 16:12:01.869691 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:01.869659 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:01.942126 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:01.942083 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:02.019771 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.019740 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:02.093131 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.093072 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:12:02.122245 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.122210 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:02.219087 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.219055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls\") pod \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " Aug 04 16:12:02.219254 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.219130 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-openshift-service-ca\") pod \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " Aug 04 16:12:02.219254 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.219165 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8cm6\" (UniqueName: \"kubernetes.io/projected/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-kube-api-access-x8cm6\") pod \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\" (UID: \"202f08dd-aaeb-4076-a61a-47e95ba9c1a9\") " Aug 04 16:12:02.219490 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.219466 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "202f08dd-aaeb-4076-a61a-47e95ba9c1a9" (UID: "202f08dd-aaeb-4076-a61a-47e95ba9c1a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:12:02.221220 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.221193 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "202f08dd-aaeb-4076-a61a-47e95ba9c1a9" (UID: "202f08dd-aaeb-4076-a61a-47e95ba9c1a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:12:02.221339 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.221253 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-kube-api-access-x8cm6" (OuterVolumeSpecName: "kube-api-access-x8cm6") pod "202f08dd-aaeb-4076-a61a-47e95ba9c1a9" (UID: "202f08dd-aaeb-4076-a61a-47e95ba9c1a9"). InnerVolumeSpecName "kube-api-access-x8cm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:12:02.320535 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.320491 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:12:02.320535 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.320531 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:12:02.320535 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.320541 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8cm6\" (UniqueName: \"kubernetes.io/projected/202f08dd-aaeb-4076-a61a-47e95ba9c1a9-kube-api-access-x8cm6\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:12:02.830349 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.830310 2559 generic.go:356] "Generic (PLEG): container finished" podID="202f08dd-aaeb-4076-a61a-47e95ba9c1a9" containerID="e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345" exitCode=0 Aug 04 16:12:02.830522 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.830361 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" Aug 04 16:12:02.830522 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.830362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" event={"ID":"202f08dd-aaeb-4076-a61a-47e95ba9c1a9","Type":"ContainerDied","Data":"e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345"} Aug 04 16:12:02.830522 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.830484 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4" event={"ID":"202f08dd-aaeb-4076-a61a-47e95ba9c1a9","Type":"ContainerDied","Data":"4944da2e834d7a9898f6d091e8d1bd17bb2b8dbc38b61736ba1afe5c0eeaa5b4"} Aug 04 16:12:02.830522 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.830512 2559 scope.go:117] "RemoveContainer" containerID="e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345" Aug 04 16:12:02.839990 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.839794 2559 scope.go:117] "RemoveContainer" containerID="e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345" Aug 04 16:12:02.840243 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:12:02.840043 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345\": container with ID starting with e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345 not found: ID does not exist" containerID="e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345" Aug 04 16:12:02.840243 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.840087 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345"} err="failed to get container status \"e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345\": rpc error: code = NotFound desc = could not find container \"e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345\": container with ID starting with e95c273a296e0fd8891c53039fcb50f3784157f5ccf196acbbe2baf889f66345 not found: ID does not exist" Aug 04 16:12:02.852520 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.852501 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4"] Aug 04 16:12:02.858790 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:02.858767 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8528c9-67464d4d98-z82n4"] Aug 04 16:12:03.289229 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:03.289198 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="202f08dd-aaeb-4076-a61a-47e95ba9c1a9" path="/var/lib/kubelet/pods/202f08dd-aaeb-4076-a61a-47e95ba9c1a9/volumes" Aug 04 16:12:05.584088 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:05.584045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:08.279971 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:08.279927 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:08.380441 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:08.380405 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:08.568950 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:08.568861 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:08.818160 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:08.818127 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:27.830829 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:27.830795 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst"] Aug 04 16:12:27.831231 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:27.831052 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" podUID="004bd657-3055-4ccb-b5d9-749cf5d64e0c" containerName="maas-api" containerID="cri-o://2c73301aa2cf042cf78de66799b469858a75fcd332c141164afb4c995749c853" gracePeriod=30 Aug 04 16:12:27.863377 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:27.863336 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:27.966598 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:27.966548 2559 generic.go:356] "Generic (PLEG): container finished" podID="004bd657-3055-4ccb-b5d9-749cf5d64e0c" containerID="2c73301aa2cf042cf78de66799b469858a75fcd332c141164afb4c995749c853" exitCode=0 Aug 04 16:12:27.966730 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:27.966619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" event={"ID":"004bd657-3055-4ccb-b5d9-749cf5d64e0c","Type":"ContainerDied","Data":"2c73301aa2cf042cf78de66799b469858a75fcd332c141164afb4c995749c853"} Aug 04 16:12:28.080306 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.080284 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:12:28.154608 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.154511 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004bd657-3055-4ccb-b5d9-749cf5d64e0c-openshift-service-ca\") pod \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " Aug 04 16:12:28.154608 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.154564 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6qwj\" (UniqueName: \"kubernetes.io/projected/004bd657-3055-4ccb-b5d9-749cf5d64e0c-kube-api-access-w6qwj\") pod \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " Aug 04 16:12:28.154845 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.154716 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004bd657-3055-4ccb-b5d9-749cf5d64e0c-maas-api-tls\") pod \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\" (UID: \"004bd657-3055-4ccb-b5d9-749cf5d64e0c\") " Aug 04 16:12:28.154929 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.154904 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004bd657-3055-4ccb-b5d9-749cf5d64e0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "004bd657-3055-4ccb-b5d9-749cf5d64e0c" (UID: "004bd657-3055-4ccb-b5d9-749cf5d64e0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:12:28.155095 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.155078 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004bd657-3055-4ccb-b5d9-749cf5d64e0c-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:12:28.156749 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.156723 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004bd657-3055-4ccb-b5d9-749cf5d64e0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "004bd657-3055-4ccb-b5d9-749cf5d64e0c" (UID: "004bd657-3055-4ccb-b5d9-749cf5d64e0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:12:28.157131 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.157111 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004bd657-3055-4ccb-b5d9-749cf5d64e0c-kube-api-access-w6qwj" (OuterVolumeSpecName: "kube-api-access-w6qwj") pod "004bd657-3055-4ccb-b5d9-749cf5d64e0c" (UID: "004bd657-3055-4ccb-b5d9-749cf5d64e0c"). InnerVolumeSpecName "kube-api-access-w6qwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:12:28.256444 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.256406 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6qwj\" (UniqueName: \"kubernetes.io/projected/004bd657-3055-4ccb-b5d9-749cf5d64e0c-kube-api-access-w6qwj\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:12:28.256444 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.256443 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004bd657-3055-4ccb-b5d9-749cf5d64e0c-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:12:28.973598 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.973537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" event={"ID":"004bd657-3055-4ccb-b5d9-749cf5d64e0c","Type":"ContainerDied","Data":"069a66f79fe4c4c91685ca45c50fc105ccf85c8f303a8411c332abcde69acd56"} Aug 04 16:12:28.974073 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.973602 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" Aug 04 16:12:28.974073 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:28.973627 2559 scope.go:117] "RemoveContainer" containerID="2c73301aa2cf042cf78de66799b469858a75fcd332c141164afb4c995749c853" Aug 04 16:12:29.000758 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:29.000734 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst"] Aug 04 16:12:29.006289 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:29.006268 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst"] Aug 04 16:12:29.289274 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:29.289199 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="004bd657-3055-4ccb-b5d9-749cf5d64e0c" path="/var/lib/kubelet/pods/004bd657-3055-4ccb-b5d9-749cf5d64e0c/volumes" Aug 04 16:12:30.350877 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:30.350846 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:30.451908 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:30.451865 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:31.042335 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:31.042288 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-8ff9cf-5bcc846544-74cst" podUID="004bd657-3055-4ccb-b5d9-749cf5d64e0c" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.67:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Aug 04 16:12:33.457702 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:33.457663 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:12:42.632211 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:12:42.632174 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:09.946061 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:09.946016 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:11.308880 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:11.308845 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:17.177450 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:17.177416 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:17.278329 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:17.278290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:17.453235 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:17.453196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:17.673306 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:17.673272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:28.348544 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.348455 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q"] Aug 04 16:13:28.349018 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.348701 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" podUID="3c3cf99b-1a93-45fb-b3f9-942c2ad31784" containerName="maas-api" containerID="cri-o://0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3" gracePeriod=30 Aug 04 16:13:28.389083 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.389049 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:28.504823 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.504780 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:28.608092 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.608047 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:28.613753 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.613734 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:13:28.693754 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.693726 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls\") pod \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " Aug 04 16:13:28.693922 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.693781 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxv4s\" (UniqueName: \"kubernetes.io/projected/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-kube-api-access-nxv4s\") pod \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " Aug 04 16:13:28.693922 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.693808 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-openshift-service-ca\") pod \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\" (UID: \"3c3cf99b-1a93-45fb-b3f9-942c2ad31784\") " Aug 04 16:13:28.694210 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.694186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c3cf99b-1a93-45fb-b3f9-942c2ad31784" (UID: "3c3cf99b-1a93-45fb-b3f9-942c2ad31784"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:13:28.695815 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.695790 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-kube-api-access-nxv4s" (OuterVolumeSpecName: "kube-api-access-nxv4s") pod "3c3cf99b-1a93-45fb-b3f9-942c2ad31784" (UID: "3c3cf99b-1a93-45fb-b3f9-942c2ad31784"). InnerVolumeSpecName "kube-api-access-nxv4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:13:28.695898 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.695836 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c3cf99b-1a93-45fb-b3f9-942c2ad31784" (UID: "3c3cf99b-1a93-45fb-b3f9-942c2ad31784"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:13:28.739955 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.739917 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:28.795186 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.795157 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:13:28.795186 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.795184 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxv4s\" (UniqueName: \"kubernetes.io/projected/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-kube-api-access-nxv4s\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:13:28.795400 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:28.795195 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cf99b-1a93-45fb-b3f9-942c2ad31784-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:13:29.134017 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.133961 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:29.279184 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.279149 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c3cf99b-1a93-45fb-b3f9-942c2ad31784" containerID="0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3" exitCode=0 Aug 04 16:13:29.279347 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.279195 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" Aug 04 16:13:29.279347 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.279231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" event={"ID":"3c3cf99b-1a93-45fb-b3f9-942c2ad31784","Type":"ContainerDied","Data":"0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3"} Aug 04 16:13:29.279347 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.279289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q" event={"ID":"3c3cf99b-1a93-45fb-b3f9-942c2ad31784","Type":"ContainerDied","Data":"63a98f26bd83244227f03e62568a714795731a2f52a5af289d72bfc15b96755e"} Aug 04 16:13:29.279347 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.279320 2559 scope.go:117] "RemoveContainer" containerID="0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3" Aug 04 16:13:29.289183 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.289165 2559 scope.go:117] "RemoveContainer" containerID="0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3" Aug 04 16:13:29.289478 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:13:29.289448 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3\": container with ID starting with 0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3 not found: ID does not exist" containerID="0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3" Aug 04 16:13:29.289659 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.289492 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3"} err="failed to get container status \"0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3\": rpc error: code = NotFound desc = could not find container \"0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3\": container with ID starting with 0940057bed81322d61dd5c2f44d122af6e7254a107a0ff5053b16708360ddca3 not found: ID does not exist" Aug 04 16:13:29.321716 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.321690 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q"] Aug 04 16:13:29.343747 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:29.343714 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b50f0-5fccb57858-8tn4q"] Aug 04 16:13:31.289975 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:31.289937 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3cf99b-1a93-45fb-b3f9-942c2ad31784" path="/var/lib/kubelet/pods/3c3cf99b-1a93-45fb-b3f9-942c2ad31784/volumes" Aug 04 16:13:35.127759 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:35.127707 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:35.244782 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:35.244745 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:35.466418 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:35.466380 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:35.667305 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:35.667268 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:46.476150 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.476117 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8"] Aug 04 16:13:46.476611 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.476367 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" podUID="294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" containerName="maas-api" containerID="cri-o://342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f" gracePeriod=30 Aug 04 16:13:46.516705 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.516678 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:46.674124 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.674089 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:46.761910 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.761885 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 16:13:46.854045 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.854012 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-openshift-service-ca\") pod \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " Aug 04 16:13:46.854254 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.854066 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls\") pod \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " Aug 04 16:13:46.854254 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.854120 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2xwh\" (UniqueName: \"kubernetes.io/projected/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-kube-api-access-v2xwh\") pod \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\" (UID: \"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9\") " Aug 04 16:13:46.854374 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.854345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" (UID: "294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:13:46.856248 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.856214 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-kube-api-access-v2xwh" (OuterVolumeSpecName: "kube-api-access-v2xwh") pod "294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" (UID: "294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9"). InnerVolumeSpecName "kube-api-access-v2xwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:13:46.856428 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.856411 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" (UID: "294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:13:46.868880 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.868849 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:46.955395 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.955356 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2xwh\" (UniqueName: \"kubernetes.io/projected/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-kube-api-access-v2xwh\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:13:46.955395 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.955396 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:13:46.955676 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:46.955410 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:13:47.369252 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.369215 2559 generic.go:356] "Generic (PLEG): container finished" podID="294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" containerID="342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f" exitCode=0 Aug 04 16:13:47.369429 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.369260 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" Aug 04 16:13:47.369429 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.369304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" event={"ID":"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9","Type":"ContainerDied","Data":"342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f"} Aug 04 16:13:47.369429 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.369355 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8" event={"ID":"294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9","Type":"ContainerDied","Data":"68c17838aa8ca4eba255551e66da9785d1318fbb5175e5a194b6e1645de909bd"} Aug 04 16:13:47.369429 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.369384 2559 scope.go:117] "RemoveContainer" containerID="342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f" Aug 04 16:13:47.370933 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.370904 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:47.378616 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.378596 2559 scope.go:117] "RemoveContainer" containerID="342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f" Aug 04 16:13:47.378866 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:13:47.378848 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f\": container with ID starting with 342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f not found: ID does not exist" containerID="342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f" Aug 04 16:13:47.378914 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.378875 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f"} err="failed to get container status \"342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f\": rpc error: code = NotFound desc = could not find container \"342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f\": container with ID starting with 342d6ea59e5242bda3534eb8fc19c2fd3c8d82b1b3997f1773767b909426651f not found: ID does not exist" Aug 04 16:13:47.431286 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.431253 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8"] Aug 04 16:13:47.444424 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:47.444395 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-67a15f-844cbcff8b-dc6c8"] Aug 04 16:13:49.297679 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:49.297645 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" path="/var/lib/kubelet/pods/294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9/volumes" Aug 04 16:13:54.110260 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:54.110215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:54.240161 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:54.240120 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:54.470195 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:54.470164 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:13:54.673153 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:13:54.673110 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:03.574942 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.574909 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c"] Aug 04 16:14:03.575394 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.575109 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" podUID="83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" containerName="maas-api" containerID="cri-o://33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429" gracePeriod=30 Aug 04 16:14:03.686308 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.686271 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" podUID="83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.65:8443/health\": dial tcp 10.132.0.65:8443: connect: connection refused" Aug 04 16:14:03.836878 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.836823 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 16:14:03.905649 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.905615 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-openshift-service-ca\") pod \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " Aug 04 16:14:03.905815 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.905702 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrbp2\" (UniqueName: \"kubernetes.io/projected/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-kube-api-access-lrbp2\") pod \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " Aug 04 16:14:03.905815 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.905725 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-maas-api-tls\") pod \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\" (UID: \"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f\") " Aug 04 16:14:03.905997 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.905966 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" (UID: "83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Aug 04 16:14:03.907816 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.907784 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" (UID: "83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Aug 04 16:14:03.907939 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:03.907883 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-kube-api-access-lrbp2" (OuterVolumeSpecName: "kube-api-access-lrbp2") pod "83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" (UID: "83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f"). InnerVolumeSpecName "kube-api-access-lrbp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Aug 04 16:14:04.006803 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.006758 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-openshift-service-ca\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:14:04.006803 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.006793 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrbp2\" (UniqueName: \"kubernetes.io/projected/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-kube-api-access-lrbp2\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:14:04.006803 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.006804 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f-maas-api-tls\") on node \"ip-10-0-138-35.ec2.internal\" DevicePath \"\"" Aug 04 16:14:04.455737 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.455706 2559 generic.go:356] "Generic (PLEG): container finished" podID="83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" containerID="33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429" exitCode=0 Aug 04 16:14:04.455944 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.455760 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" Aug 04 16:14:04.455944 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.455755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" event={"ID":"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f","Type":"ContainerDied","Data":"33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429"} Aug 04 16:14:04.455944 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.455818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c" event={"ID":"83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f","Type":"ContainerDied","Data":"4725e630269833402d002c22b9b3b094db2cb3838e86de055cac5c0fbfe25eca"} Aug 04 16:14:04.455944 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.455843 2559 scope.go:117] "RemoveContainer" containerID="33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429" Aug 04 16:14:04.468138 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.468114 2559 scope.go:117] "RemoveContainer" containerID="33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429" Aug 04 16:14:04.468419 ip-10-0-138-35 kubenswrapper[2559]: E0804 16:14:04.468397 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429\": container with ID starting with 33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429 not found: ID does not exist" containerID="33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429" Aug 04 16:14:04.468501 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.468429 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429"} err="failed to get container status \"33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429\": rpc error: code = NotFound desc = could not find container \"33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429\": container with ID starting with 33e96032a3b0761c98dc4781220614877acf3001e17e5ad068e912fc1280e429 not found: ID does not exist" Aug 04 16:14:04.485113 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.485089 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c"] Aug 04 16:14:04.496735 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.496710 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4d9499-75fc6b688d-4gd8c"] Aug 04 16:14:04.738485 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.738392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:04.921693 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:04.921663 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:05.289047 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:05.289015 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" path="/var/lib/kubelet/pods/83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f/volumes" Aug 04 16:14:10.647453 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:10.647419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:10.748910 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:10.748842 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:10.941082 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:10.940996 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:11.580981 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:11.580949 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b59d4dc5c-b5fkg_9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a/authorino/0.log" Aug 04 16:14:11.836375 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:11.836292 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:17.399953 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:17.399921 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vvhzh_d08d6054-e6eb-47ad-9ab7-42a5486668af/manager/0.log" Aug 04 16:14:17.691445 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:17.691361 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78bcb9f478-7v569_aa61e7b9-3ca3-42cf-bd78-59f3d7f17216/manager/0.log" Aug 04 16:14:17.811199 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:17.811165 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bd7pk_1747d4d9-7474-45ae-8b9f-0889f79b22cd/server/0.log" Aug 04 16:14:17.945451 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:17.945412 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jtsss_244b7eea-6747-4456-99d2-c82257b5bf6e/manager/0.log" Aug 04 16:14:18.224993 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:18.224922 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77c9c696b6-dhhvz_f2d73a7a-cd9f-460a-b470-4ea1a3f66292/copy-manifests/0.log" Aug 04 16:14:18.232677 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:18.232652 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77c9c696b6-dhhvz_f2d73a7a-cd9f-460a-b470-4ea1a3f66292/manager/0.log" Aug 04 16:14:18.780782 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:18.780755 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bf88d8cb-9mzrj_28f65295-fdcd-47a2-a3dc-78809c2219c0/maas-api/0.log" Aug 04 16:14:19.261482 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:19.261442 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-cdkpk_e0667036-68a7-44d4-ba07-2260ddd37fd7/postgres/0.log" Aug 04 16:14:20.118435 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:20.118323 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b59d4dc5c-b5fkg_9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a/authorino/0.log" Aug 04 16:14:20.246445 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:20.246416 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-q4vw5_a7a0fe87-bd13-424d-b988-9d10c5801af5/manager/0.log" Aug 04 16:14:20.369089 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:20.369009 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-jq74f_0ea2134c-e8d7-4dd1-99b4-c75708169752/manager/0.log" Aug 04 16:14:20.617544 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:20.617513 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-95p42_d835dfbd-22a6-41df-872d-5f7652e0a83f/manager/0.log" Aug 04 16:14:20.865482 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:20.865449 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-r92wl_1df49bf1-def8-4a17-8d71-8a567366f1b0/manager/0.log" Aug 04 16:14:21.476430 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:21.476400 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9dff769-mctsj_309833bf-fc35-445a-9c48-c31ccd4c1f2d/kube-auth-proxy/0.log" Aug 04 16:14:21.942739 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:21.942712 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pzfkj_3494aa11-cbd1-4c2f-bae2-b565ef4d1f80/payload-processing/0.log" Aug 04 16:14:22.063666 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:22.063637 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59fcd9fd6d-l6gbg_c330d986-f440-4bb4-b152-bdb21af2925e/router/0.log" Aug 04 16:14:22.428468 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:22.428391 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-8q9wm_5aed4231-9239-406f-8caa-9c897350e55c/main/0.log" Aug 04 16:14:22.788976 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:22.788949 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn245g_21092b43-962a-4beb-8b47-1c16bf3ff516/main/0.log" Aug 04 16:14:29.115395 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:29.115357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:29.262076 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:29.262032 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" limit=1024 Aug 04 16:14:34.246702 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:34.246668 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6nqlc_e629fc59-eefc-49cd-9bb5-9a329b04633b/global-pull-secret-syncer/0.log" Aug 04 16:14:34.615522 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:34.615437 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4nfzb_0a1d41dd-149a-4de6-a3e7-993833e83ac3/konnectivity-agent/0.log" Aug 04 16:14:34.968919 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:34.968886 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-35.ec2.internal_14b643114c2bf06208c77f7bdab0056d/haproxy/0.log" Aug 04 16:14:39.350721 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:39.350688 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b59d4dc5c-b5fkg_9d6ff147-a6f7-4d4e-9e3e-3e9c814e885a/authorino/0.log" Aug 04 16:14:39.384842 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:39.384806 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-q4vw5_a7a0fe87-bd13-424d-b988-9d10c5801af5/manager/0.log" Aug 04 16:14:39.426353 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:39.426325 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-jq74f_0ea2134c-e8d7-4dd1-99b4-c75708169752/manager/0.log" Aug 04 16:14:39.593731 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:39.593700 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-68cf589589-95p42_d835dfbd-22a6-41df-872d-5f7652e0a83f/manager/0.log" Aug 04 16:14:39.771456 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:39.771425 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-r92wl_1df49bf1-def8-4a17-8d71-8a567366f1b0/manager/0.log" Aug 04 16:14:41.572819 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:41.572786 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-sxq7z_f2fb7ca7-9668-4869-a3df-5022e5c62cf1/cluster-monitoring-operator/0.log" Aug 04 16:14:42.109382 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:42.109350 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwn2f_3764a934-28c3-453f-90f2-d2e3fb7ea247/node-exporter/0.log" Aug 04 16:14:42.144100 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:42.144074 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwn2f_3764a934-28c3-453f-90f2-d2e3fb7ea247/kube-rbac-proxy/0.log" Aug 04 16:14:42.175433 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:42.175412 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwn2f_3764a934-28c3-453f-90f2-d2e3fb7ea247/init-textfile/0.log" Aug 04 16:14:44.134147 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.134113 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-4tkkk_55a38cc2-ffaa-46be-a929-a3d7fb71250d/networking-console-plugin/0.log" Aug 04 16:14:44.521504 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521482 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5"] Aug 04 16:14:44.521900 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521888 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" containerName="maas-api" Aug 04 16:14:44.521945 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521902 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" containerName="maas-api" Aug 04 16:14:44.521945 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521914 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3cf99b-1a93-45fb-b3f9-942c2ad31784" containerName="maas-api" Aug 04 16:14:44.521945 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521920 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3cf99b-1a93-45fb-b3f9-942c2ad31784" containerName="maas-api" Aug 04 16:14:44.521945 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521937 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" containerName="maas-api" Aug 04 16:14:44.521945 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521943 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521949 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="004bd657-3055-4ccb-b5d9-749cf5d64e0c" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521955 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="004bd657-3055-4ccb-b5d9-749cf5d64e0c" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521965 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="202f08dd-aaeb-4076-a61a-47e95ba9c1a9" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.521971 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="202f08dd-aaeb-4076-a61a-47e95ba9c1a9" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.522026 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="202f08dd-aaeb-4076-a61a-47e95ba9c1a9" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.522034 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3cf99b-1a93-45fb-b3f9-942c2ad31784" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.522040 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="83cbaa90-8ed8-4a9d-9ca0-cb332fc5260f" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.522045 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="004bd657-3055-4ccb-b5d9-749cf5d64e0c" containerName="maas-api" Aug 04 16:14:44.522101 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.522051 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="294a5f48-4e0a-4dc9-bad5-6ea47c79c6f9" containerName="maas-api" Aug 04 16:14:44.526413 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.526398 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.528953 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.528928 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtx9z\"/\"kube-root-ca.crt\"" Aug 04 16:14:44.529114 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.529095 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qtx9z\"/\"openshift-service-ca.crt\"" Aug 04 16:14:44.529916 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.529899 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qtx9z\"/\"default-dockercfg-65jnv\"" Aug 04 16:14:44.535399 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.535376 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5"] Aug 04 16:14:44.557729 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.557706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-lib-modules\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.557832 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.557780 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-sys\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.557878 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.557827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-proc\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.557878 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.557870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-podres\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.557947 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.557931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjq5\" (UniqueName: \"kubernetes.io/projected/55f39875-e255-444f-b470-5aa341557a54-kube-api-access-4bjq5\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659228 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-sys\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659381 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-proc\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659381 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-podres\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659381 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-sys\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659381 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjq5\" (UniqueName: \"kubernetes.io/projected/55f39875-e255-444f-b470-5aa341557a54-kube-api-access-4bjq5\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659381 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-lib-modules\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659381 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-proc\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659747 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-podres\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.659747 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.659489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55f39875-e255-444f-b470-5aa341557a54-lib-modules\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.680234 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.680210 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjq5\" (UniqueName: \"kubernetes.io/projected/55f39875-e255-444f-b470-5aa341557a54-kube-api-access-4bjq5\") pod \"perf-node-gather-daemonset-r2vq5\" (UID: \"55f39875-e255-444f-b470-5aa341557a54\") " pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.836694 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.836604 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:44.845193 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.845168 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/2.log" Aug 04 16:14:44.855282 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.855259 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-9j44d_814798a3-c8db-4727-920f-dfca65127d18/console-operator/3.log" Aug 04 16:14:44.974098 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:44.974074 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5"] Aug 04 16:14:44.976163 ip-10-0-138-35 kubenswrapper[2559]: W0804 16:14:44.976137 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod55f39875_e255_444f_b470_5aa341557a54.slice/crio-cc32a9b30d6d644eb35d810301eab6e87119722b48c9e794d7c73fc0ce93b2b1 WatchSource:0}: Error finding container cc32a9b30d6d644eb35d810301eab6e87119722b48c9e794d7c73fc0ce93b2b1: Status 404 returned error can't find the container with id cc32a9b30d6d644eb35d810301eab6e87119722b48c9e794d7c73fc0ce93b2b1 Aug 04 16:14:45.666568 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:45.666534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" event={"ID":"55f39875-e255-444f-b470-5aa341557a54","Type":"ContainerStarted","Data":"b29557ff0ab63e2b35cfb2d708959ace832657817b5ac85ef21976f9265117fd"} Aug 04 16:14:45.666974 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:45.666604 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" event={"ID":"55f39875-e255-444f-b470-5aa341557a54","Type":"ContainerStarted","Data":"cc32a9b30d6d644eb35d810301eab6e87119722b48c9e794d7c73fc0ce93b2b1"} Aug 04 16:14:45.666974 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:45.666628 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:45.687272 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:45.687223 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" podStartSLOduration=1.687206566 podStartE2EDuration="1.687206566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 16:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 16:14:45.685715699 +0000 UTC m=+3851.039803459" watchObservedRunningTime="2026-08-04 16:14:45.687206566 +0000 UTC m=+3851.041294324" Aug 04 16:14:46.242635 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:46.242601 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-k789b_dc1eb654-576c-44bd-ab10-f7a2ed3d75e3/volume-data-source-validator/0.log" Aug 04 16:14:47.477341 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:47.477310 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bng9_edc4b4c4-6033-4d2d-87ff-c1be81027a01/dns/0.log" Aug 04 16:14:47.499602 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:47.499562 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2bng9_edc4b4c4-6033-4d2d-87ff-c1be81027a01/kube-rbac-proxy/0.log" Aug 04 16:14:47.707877 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:47.707847 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j8h9m_7632fc34-ddbf-4d83-8048-0a9d86cdf1a0/dns-node-resolver/0.log" Aug 04 16:14:48.373858 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:48.373826 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lk76r_fa1d177c-d3ac-425d-a836-58dc1dc7b425/node-ca/0.log" Aug 04 16:14:49.643283 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:49.643245 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9dff769-mctsj_309833bf-fc35-445a-9c48-c31ccd4c1f2d/kube-auth-proxy/0.log" Aug 04 16:14:49.809401 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:49.809370 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pzfkj_3494aa11-cbd1-4c2f-bae2-b565ef4d1f80/payload-processing/0.log" Aug 04 16:14:49.846594 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:49.846560 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59fcd9fd6d-l6gbg_c330d986-f440-4bb4-b152-bdb21af2925e/router/0.log" Aug 04 16:14:50.799378 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:50.799349 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mlgv6_c389923e-0891-45d8-b662-1700d98d597b/serve-healthcheck-canary/0.log" Aug 04 16:14:51.384189 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:51.384155 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-z5kg4_e8b23c6f-dabe-4c2d-9993-e7682fdb4c26/insights-operator/0.log" Aug 04 16:14:51.385698 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:51.385677 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-z5kg4_e8b23c6f-dabe-4c2d-9993-e7682fdb4c26/insights-operator/1.log" Aug 04 16:14:51.573682 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:51.573653 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mg2d6_b3f2ae45-d92a-4422-9ecb-42a0c822daf6/kube-rbac-proxy/0.log" Aug 04 16:14:51.598744 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:51.598720 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mg2d6_b3f2ae45-d92a-4422-9ecb-42a0c822daf6/exporter/0.log" Aug 04 16:14:51.637473 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:51.637395 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mg2d6_b3f2ae45-d92a-4422-9ecb-42a0c822daf6/extractor/0.log" Aug 04 16:14:51.680980 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:51.680958 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qtx9z/perf-node-gather-daemonset-r2vq5" Aug 04 16:14:54.272201 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:54.272173 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vvhzh_d08d6054-e6eb-47ad-9ab7-42a5486668af/manager/0.log" Aug 04 16:14:54.856761 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:54.856675 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78bcb9f478-7v569_aa61e7b9-3ca3-42cf-bd78-59f3d7f17216/manager/0.log" Aug 04 16:14:54.905719 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:54.905690 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-bd7pk_1747d4d9-7474-45ae-8b9f-0889f79b22cd/server/0.log" Aug 04 16:14:55.034197 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:55.034169 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-jtsss_244b7eea-6747-4456-99d2-c82257b5bf6e/manager/0.log" Aug 04 16:14:55.212246 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:55.212212 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77c9c696b6-dhhvz_f2d73a7a-cd9f-460a-b470-4ea1a3f66292/manager/0.log" Aug 04 16:14:55.244176 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:55.244146 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77c9c696b6-dhhvz_f2d73a7a-cd9f-460a-b470-4ea1a3f66292/copy-manifests/0.log" Aug 04 16:14:55.714360 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:55.714328 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9dff769-mctsj_309833bf-fc35-445a-9c48-c31ccd4c1f2d/kube-auth-proxy/0.log" Aug 04 16:14:55.822328 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:55.822299 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pzfkj_3494aa11-cbd1-4c2f-bae2-b565ef4d1f80/payload-processing/0.log" Aug 04 16:14:55.844212 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:55.844182 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59fcd9fd6d-l6gbg_c330d986-f440-4bb4-b152-bdb21af2925e/router/0.log" Aug 04 16:14:57.566988 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:57.566908 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86b9dff769-mctsj_309833bf-fc35-445a-9c48-c31ccd4c1f2d/kube-auth-proxy/0.log" Aug 04 16:14:57.686185 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:57.686155 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-pzfkj_3494aa11-cbd1-4c2f-bae2-b565ef4d1f80/payload-processing/0.log" Aug 04 16:14:57.716168 ip-10-0-138-35 kubenswrapper[2559]: I0804 16:14:57.716132 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59fcd9fd6d-l6gbg_c330d986-f440-4bb4-b152-bdb21af2925e/router/0.log"